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ABSTRACT 


The  nature  of  warfare  has  changed  dramatically  during  the  last  decade.  Western 
annies  are  increasingly  required  to  conduct  complex  operations  in  urban  terrain  against 
asymmetric  threats.  These  opponents  use  cities  and  their  inhabitants  for  cover  and 
concealment.  In  such  situations,  modern  equipped  armies  often  cannot  fully  utilize  many 
of  their  most  powerful  weapons.  To  overcome  this  situation,  modern  communication 
systems  are  being  acquired  and  deployed  to  provide  real-time  reconnaissance;  thereby, 
attempting  to  neutralize  the  threat  through  enhanced  situational  awareness.  This  research 
addresses  the  potential  impacts  of  communication  from  airborne  sensors  on  assisting  a 
convoy  in  finding  its  way  through  a  hostile  city  quarter  (based  on  Mazar-E-Sharif, 
Afghanistan)  in  which  militia  forces  try  to  interdict  them  via  street  blockades  and 
ambushes.  The  implementation  is  done  in  the  agent-based  simulation  Map  Aware  Non- 
Uniform  Automata  (MANA).  The  results  show  that  the  current  MANA  version  is  not 
sufficiently  capable  to  handle  routing  problems  in  urban  terrain.  Specifically,  the 
movement  algorithm  is  “locally  greedy”  and  not  flexible  enough  to  project  into  the 
future — as  real  human  decision  makers  do.  Many  workarounds  were  developed  to 
mitigate  this  limitation.  The  analysis  shows  that  the  number  of  blockades  is  the  single 
most  important  factor  in  determining  the  convoy’s  success.  Of  the  communication 
factors,  network  latency  has  the  most  impact.  For  the  convoy  to  effectively  use  the 
information,  it  needs  to  get  from  the  sensor  to  the  convoy  in  1 1  seconds. 
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EXECUTIVE  SUMMARY 


The  nature  of  warfare  has  changed  dramatically  during  the  last  decade.  Western 
annies  are  increasingly  required  to  conduct  complex  operations  in  urban  terrain  against 
asymmetric  threats.  These  opponents  use  cities  and  their  inhabitants  for  cover  and 
concealment.  In  such  situations,  modern  equipped  armies  often  cannot  fully  utilize  many 
of  their  most  powerful  weapons.  To  overcome  this  situation,  modern  communication 
systems  are  being  acquired  and  deployed  to  provide  real-time  reconnaissance,  thereby 
attempting  to  neutralize  the  threat  through  enhanced  situational  awareness.  This  research 
addresses  the  potential  impacts  of  communication  from  airborne  sensors  on  assisting  a 
convoy  in  finding  its  way  through  a  hostile  city  quarter  in  which  militia  forces  try  and 
interdict  them  via  street  blockades  and  ambushes. 

The  vignette  chosen  to  explore  the  communication  aspects  in  urban  combat  is 
inspired  by  the  “Black  Hawk  Down”  incident  in  Mogadishu,  Somalia.  As  depicted  in  the 
movie,  a  convoy  has  to  make  its  way  through  a  hostile  city  neighborhood  in  which  militia 
forces  build  blockades  and  try  to  ambush  the  convoy.  Unfortunately  for  the  convoy,  in 
Mogadishu,  the  delay  in  getting  infonnation  from  their  airborne  reconnaissance  platfonns 
(helicopters)  made  the  information  almost  useless.  In  our  situation,  the  convoy  is  trying 
to  navigate  through  terrain  based  on  Mazar-E-Sharif,  Afghanistan.  The  airborne  sensor 
grid  supporting  the  convoy  consists  of  hovering  Unmanned  Aerial  Vehicles  (UAVs). 
The  UAVs  provide  near  real-time  infonnation  about  blockades. 

The  scenario  is  implemented  as  an  agent-based  simulation  using  the  software 
package  Map  Aware  Non-Uniform  Automata  (MANA),  see  Figure  1.  However, 
transfening  a  classic  network  routing  problem  into  an  agent-based  model  is  not  easily 
done.  The  software’s  limitations  make  it  necessary  to  use  many  workarounds.  In 
particular,  the  convoy’s  movement  decisions — which  MANA  limits  to  greedy  local 
choices — have  to  be  made  more  robust.  The  partial  screen  shot  (Figure  2)  shows  the 
main  ideas  in  our  approach  to  making  the  convoy  behave  reasonably. 
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Figure  1.  The  original  map  of  Mazar-E-Sharif  and  the  MANA  terrain  map. 


Figure  2.  Helper  agents  used  to  assist  the  convoy  in  utilizing  information.  (Best 
viewed  in  color) 

The  convoy  can  rely  on  a  grid  of  friends,  comparable  to  the  street  map.  These 
friends  draw  the  convoy  into  usable  streets  (green)  or  warn  it  and  repel  it  when  the  street 
is  blocked  (yellow).  Having  additional  information  from  the  sensor  grid,  the  convoy  is 
informed  by  these  ‘logic  helpers’  on  which  roads  are  clear.  An  important  question  is, 
how  do  these  workarounds  influence  the  model’s  quality  and  usability? 

In  order  to  settle  the  analysis  on  a  broader  base,  the  factors  are  divided  into 
uncontrollable  and  controllable  sets.  The  factors  the  convoy  cannot  control  include  the 
number  and  placement  of  blockades.  The  controllable  factors  are  those  associated  with 
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its  sensors  and  communications  network.  These  include  latency,  reliability,  capacity, 
sensor  range,  and  update  rate.  Nearly  Orthogonal  Latin  Hypercubes  are  used  to 
efficiently  conduct  the  experiments. 

Two  experimental  sets  are  conducted.  The  first  experiments  help  assess  the 
model’s  quality  by  comparing  the  convoy’s  ability  to  successfully  reach  its  goal  under  25 
blockade  combinations,  both  with  communications  and  without  communications.  That 
is,  without  communications  the  convoy  must  rely  only  on  its  own  visual  sensor;  with 
communications  the  convoy  can  also  take  advantage  of  the  information  given  to  it  by  a 
grid  of  hovering  UAVs.  The  measure  used  to  quantify  the  effectiveness  of  the  convoy  in 
traveling  to  its  objective  is  the  number  of  hits  the  convoy  suffers.  The  main  result  from 
the  quality  assessing  experiment  is  shown  in  Figure  3. 


Oveihiy  Plot 


^combination 

x  — Equal&Better  n  — Equal  ❖  — Better 

Figure  3.  Proportion  of  times  the  convoy  with  communication  does  better,  equal,  or 
equal  &  better  than  the  convoy  without  communications  for  25  separate 
combinations  of  blockades.  The  number  of  blockades  increases  with  the 
x-axis.  (Best  viewed  in  color) 


The  experiment  was  run  to  compare  the  convoy’s  behavior  with  and  without 
communications.  The  graphical  analysis  reveals  that  for  a  large  number  of  blockades  the 
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convoy  with  communications  does  better  only  about  60  percent  of  the  time — a 
disappointingly  low  number.  On  the  other  hand,  the  statistical  analysis  shows  that 
communications  has  an  improving  impact  by  reducing  the  mean  number  of  hits  the 
convoy  suffers  for  all  25  blockade  configurations. 

The  second  experiment,  the  main  focus  of  this  thesis,  examines  the 
communications  factors.  The  25  combinations  of  blockades  are  tested  against  65 
communication  settings.  This  experiment  contains  1625  input  combinations,  with  100 
replications  each.  The  analysis  of  the  output  data  from  these  162,500  computational 
experiments  shows  that  the  number  of  blockades  has  the  most  impact  on  the  convoy’s 
success.  Unsurprisingly,  latency  within  a  communication  network  is  the  most  important 
communications  factor,  followed  by  the  pause  (delay  time)  between  the  information 
updates  given  by  the  sensor  grid.  That  is,  even  good  information  in  such  a  situation  has 
to  be  timely,  or  it  is  nearly  worthless.  The  exclusion  of  other  factors,  such  as  reliability 
and  capacity,  must  be  taken  in  the  context  of  our  static  scenario — i.e.,  there  are  no 
moving  enemies.  Figure  4  shows  part  of  a  partition  tree  on  the  communications  factors. 


Figure  4.  A  portion  of  the  regression  tree  with  the  partitions  made  only  on 
communications  factors.  The  times  are  in  seconds.  We  see  the 
importance  of  the  hovering  sensors  getting  infonnation  quickly  to  the 
convoy,  and  its  effect  on  the  mean  number  of  hits  the  convoy  suffers. 
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Other  lessons  learned  and  key  takeaways  from  implementing  this  scenario  in 
MANA  include: 

•  The  current  simulation  software  complicates  the  realistic  modeling  of 
movements  in  urban  terrain,  and  will  continue  to  do  so  as  long  as  a  routing 
problem  is  involved.  The  necessity  of  an  underlying  network  structure  for 
movement  decisions  became  obvious  to  the  author. 

•  The  communication  links  provided  by  the  software  package  do  not  reflect 
the  current  technology  standards.  In  order  to  model  realistic  digital 
communication  networks,  workarounds  had  to  be  used.  These 
workarounds  reduced  the  simplicity  and  the  idea  of  agent-based  models. 
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I.  INTRODUCTION 


A.  BACKGROUND 

A  military  unprepared  for  urban  operations  across  a  broad  spectrum  is 

unprepared  for  tomorrow.  [Peters  1996] 

Since  the  fall  of  the  iron  curtain  in  1990  and  the  end  of  the  Cold  War,  warfare  has 
changed  drastically.  The  old  doctrines  were  modeled  on  the  expectation  that  nearly  all 
decisive  battles  would  take  place  in  open  terrain.  Both  combatants  in  the  Cold  War — 
NATO  and  those  bound  by  the  Warsaw  Pact — clearly  desired  to  avoid  combat  in  urban 
environments.  Behind  this  intention  to  steer  conflict  away  from  cities  was  the  possibility 
of  an  incalculable  number  of  casualties  in  urban  zones. 

On  the  other  hand,  “control  of  cities  always  has  been  vital  to  military  success, 
practically  and  symbolically.”1  Today,  Western  armies  face  more  and  more  asymmetric 
threats  in  which  the  opponent  attempts  to  draw  the  forces  into  an  urban  environment. 
These  opponents  are  familiar  with  the  area  and  many  of  the  high-technology  weapons  of 
the  West  are  largely  ineffective.  Furthermore,  opponents  use  civilians  in  the  cities  as 
shields  and  restrict  the  engagement  of  weapon  systems  which  are  designed  for  open  space 
combat.  The  current  situation  in  Baghdad  illustrates  the  problems  that  Western  forces 
encounter  in  this  new  urban  combat.  American  political  and  military  leaders  are  clearly 
focused  on  securing  the  city.  In  fact,  success  in  this  major  city  will  likely  determine  the 
success  of  the  coalition  forces  in  all  of  Iraq.  In  recent  months,  however,  terrorist 
organizations  have  attempted  to  destabilize  Baghdad. 

The  likelihood  of  troop  engagement  in  cities  is  increasing.  One  factor  in  this  new 
development,  as  mentioned  before,  is  the  opponent’s  strategy  to  use  the  advantage  of 
familiar  urban  terrain  and  the  cover  of  non-combatants  in  the  streets.  On  the  other  hand,  a 
modem  equipped  Western  army  lacks  the  opportunity  to  unfold  its  total  power.  As  a 
result,  this  army  experiences  a  reduced  effectiveness  of  high-tech  weapon  systems  and 
limited  communications.  Another  aspect  of  increases  in  urban  combat  is  the  nature  of 


1  Ralph  Peters,  “Our  Soldiers,  Their  Cities,”  Parameters,  Spring  1996,  pp.  43-50. 
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cities  themselves.  Demographic  data  shows  that  urbanization  is  an  ongoing  process. 
Cities  act  like  a  center  of  gravity,  attracting  people  to  move  there.  Currently,  half  of  the 
world’s  population  lives  in  and  around  cities,  and  a  prognosis  for  the  next  century  shows 
that  about  70  percent  of  people  will  make  their  homes  in  an  urbanized  terrain.2 

Even  in  low-intensity  peacemaking  and  peacekeeping  missions  like  NATO  led 
KFOR  in  Kosovo,  forces  must  be  deployed  in  urban  environments.  “Operating  in  a 
potentially  hostile  city  is  every  soldier’s  nightmare.”3  Behind  each  corner,  out  of  every 
window,  the  enemy  can  attack  the  patrolling  soldiers.  These  kinds  of  operations  are 
challenging  for  a  modern  anny.  In  a  training  mission  conducted  in  Oakland,  California, 
U.S  Marines  attacked  an  Old  Navy  Hospital  and  suffered  high  losses  (70%)  during  the 
first  hours  of  combat.4  At  the  time,  most  Western  armies  were  ill-prepared  for  urban 
operations. 

In  order  to  overcome  this  miserable  situation,  military  organizations  have  recently 
made  several  changes  in  the  preparation  and  training  of  their  soldiers.  Physical  and 
psychological  training  are  necessary  in  order  that  the  soldier  may  avoid  distraction  caused 
by  a  totally  destroyed  environment  and  the  human  misery  caused  by  war.  The  militaries 
have  also  improved  individual  equipment.  Modem  modular  gear  contains  night  vision 
goggles  and  a  rifle  with  laser  ranger,  compass  and  head-ups  display. 

The  key  factor  to  increase  survivability  in  urban  combat,  however,  is  and  will  be 
information  sharing  and  collaboration.  In  a  network-centric  operation,  each  soldier 
benefits  from  precise  information.  Consequently,  the  situational  awareness,  speed  of 
command  and  mission  effectiveness  improves  dramatically.5 


2  Jean  Kumagai,  “Fighting  in  Streets,”  IEEE  Spectrum,  February  2001. 

3  Gerald  Yonas  and  Timothy  Moy,  “Emerging  Technologies  and  Military  Operations  in  Urban 
Terrain,”  in  Michael  C.  Desch  “Soldiers  In  Cities;  Military  Operations  On  Urban  Terrain,”  2001. 

4  Jean  Kumagai,  “Fighting  in  Streets,”  IEEE  Spectrum,  February  2001. 

5  Definition  of  Network-centric  operations  by  MORS  online,  www.mors.org,  October  2006. 
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“Real-time  and  detailed  information  is  critical  to  the  success  of  ground  combat 
forces.”6  Over-the-hill-reconnaissance  done  by  a  networked,  unmanned  aerial  sensor 
system  provides  the  commanding  officer  pertinent  infonnation.  Thus,  troops  can  cover 
more  area  and  gain  situational  awareness  to  aid  them  in  the  hostile  environment. 

Successful  reconnaissance  systems  will  therefore  meet  the  following 
requirements:7 

•  Detect,  identify,  and  track  enemy  targets, 

•  Be  equipped  with  robust,  secure  and  stealthy  sensor  arrays  and 

•  Contain  jam  resistant  links  for  sensor  fusion  and  command  and  control. 
Furthermore,  in  order  to  reach  the  commanding  officer  or  fighting  soldier  in  real-time, 
data  processing  from  the  sensor  to  command  and  control  must  be  automated.  Challenges 
such  as  the  amount  of  collected  data  and  making  comparisons  with  databases  to 
recognize  and  classify  an  enemy  correctly  are  expected  to  be  solvable  with  technology. 
By  automating  these  data  intensive  procedures,  human  intervention  would  normally  be 
excluded  from  this  process  and  thus  would  reduce  latency. 

B.  GERMANY’S  C4ISR  APPROACH 

At  this  time,  the  German  Armed  Forces  does  not  possess  fully  networked 
capabilities.  They  have  recognized,  however,  that  C4ISR  (command,  control, 
communications,  computers,  intelligence,  surveillance  and  reconnaissance)  is  the  heart  of 
modem  warfare.8 

Efforts  to  achieve  a  network-centric  military  are  often  slowed  by  budget  concerns. 
The  lion’s  share  of  the  Gennan  military  procurement  budget  is  spent  on  a  few  major 
acquisitions  (e.g.,  Eurofighter,  air-transporter  AIRBUS  400M,  Tiger  and  NH- 
helicopters).  The  Gennan  military  must  purchase  all  of  these  investments  in  fulfillment 
of  long-binding  contracts  with  German  and  European  companies. 

6  Gene  Klager,  “Networked  Sensors  for  the  Combat  Forces,'”  in  “Unmanned/Unattended  Sensors  and 
Sensor  Networks,”  Edward  M.  Carapezza  (Editor),  November  2004,  pp.  204-214. 

7  Jay  Nemeroff,  Luis  Garcia,  Dan  Hampel,  Stef  DiPierro,  “Application  of  Sensor  Network 
Communications,”  IEEE  2001. 

8  Gordon  Adams,  Guy  ben-Ari,  John  Logsdon,  Ray  Williamson,  “Bridging  the  Gap,”  October  2004. 
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Another  obstacle  to  progress  in  networking  capability  is  the  lack  of  inter¬ 
operability  among  existing  C3  equipment.  For  example,  HEROS  (Army  command  and 
control  system  for  digitally-supported  command  operations  in  staffs)  is  introduced  to 
corps,  divisions  and  brigades.  This  system  has  no  interface  to  the  lower  echelon  system 
FAUST  (command  and  control  provision)  for  regiments  and  battalions. 

The  FAUST  system  is  mounted  on  most  of  the  vehicles  and  tanks  operating  in 
Afghanistan  and  Kosovo  and  delivers  a  variety  of  support  for  the  forces.  Its  updated  map 
mirrors  the  friendly  forces’  positions  and  the  reconnaissance  results  about  the  enemy  that 
have  been  entered  manually.  Other  tools  are  also  helpful,  such  as  emergency  call,  free 
text  messages  and  automated  updates  and  reports  concerning  the  position  of  the  vehicle. 

The  German  Army  has  one  successful  UAV  at  its  disposal.  The  drone  CL-289  is 
able  to  take  black/white  or  infrared  pictures  and  deliver  real-time  video  data.  These 
results  must  be  analyzed  by  operators  and  entered  into  the  C3  software  packages.  Other 
UAV  programs  have  been  abandoned  during  recent  years  due  to  budget  limitations. 

The  German  Anny  will  attempt  to  overcome  their  technological  handicap  by 
achieving  interoperability  between  the  C3  systems  and  with  other  branches  and  allies. 
The  automated  data  entries  are  absolutely  necessary  to  gain  a  real-time  information  flow 
in  an  ongoing  operation. 

New  UAV  programs  have  been  launched  to  improve  over-the-hill  reconnaissance 
capabilities  with  the  forces.  One  such  program  has  resulted  in  the  FANCOPTER  (shown 
in  Figure  5),  developed  by  EMT.  The  FANCOPTER  is  a  small  hovering  UAV  specially 
designed  for  missions  in  urban  terrain  or  indoors.  With  a  rotor  diameter  of  half  a  meter, 
the  UAV  stabilizes  in  a  position  by  using  acceleration  sensors  which  measure  the  wind 
speed.  The  FANCOPTER  may  be  equipped  with  different  camera  types,  including 
digital,  video,  or  IR  camera.  At  this  stage  of  development,  its  radius  of  action  is  about 
500  meters,  and  the  maximum  mission  length  is  about  20  minutes. 
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Figure  5.  FANCOPTER  developed  by  EMT  Penzberg.  (From:  www.emt- 
penzberg.de  (August  2006)). 

A  fully  functional  C4ISR  system  is  essential  to  the  German  Army.  Since  the 
reunification  in  October  1990,  the  German  Armed  Forces  have  begun  to  support  United 
Nations  missions.  As  a  NATO  member,  Germany  also  took  part  in  the  Kosovo  invasion 
after  the  air  strikes  in  1999. 

For  the  last  four  years,  Germany  has  contributed  a  large  contingent  to  the 
stabilization  forces  in  Afghanistan  (ISAF).  The  latest  news  shows  that  the  war  on 
terrorism  has  not  yet  been  won  in  this  region.  The  threat  to  German  soldiers  in  this 
mission  abroad  is  constantly  increasing. 

With  the  transfer  of  the  main  contingent  from  Kabul  to  Mazar-E-Sharif 
completed,  Germany  has  taken  over  the  responsibility  for  the  Northern  Province.  A  map 
of  Mazar-E-Sharif  is  used  for  the  vignette  presented  later  in  this  thesis. 

The  vignette  in  this  thesis  is  modeled  in  MANA.  This  agent-based  simulation  is 
well-accepted  in  the  simulation  community.  The  scenario  is  also  implemented  in  IT- 
SimBw.  Besides  the  development  and  procurement  of  modem  communication 
technology,  “The  German  Armed  Forces  Federal  Office  for  Information  Management 
and  Information  Technology,”  in  cooperation  with  Fraunhoffer  Institute,  started  to  build 
IT-SimBw  as  a  new  agent-based  simulation  tool  that  will  examine  communication 
aspects  of  combat.  The  first  demonstrations  of  this  new  software  package  were  done 
during  the  Project  Albert  workshops  in  Honolulu  in  February  2006  and  Boppard, 
Germany  in  June  2006.  Modeling  insights  experienced  during  this  research  have  proved 
helpful  to  the  developers  of  IT-SimBw.  Similarly,  some  limitations  of  the  widely  used 
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“simulation  software  MANA  have  become  apparent  during  the  course  of  research 
conducted  for  this  thesis.  It  is  not  within  the  scope  of  this  thesis,  however,  to  make  a 
comparison  between  these  simulation  packages. 


C.  SCOPE 

The  thesis  research  operates  on  the  basic  assumption  that  the  employed  forces 
have  a  modern  C4ISR  system  available  for  a  specific  mission.  The  listed  questions  of 
interest  are  addressed  to  explore  the  communication  aspects  of  fighting  in  urban  terrain: 

•  What  are  the  impacts  of  various  factors  (latency,  reliability,  data  updating 
frequency,  capacity  and  sensor  ranges)  for  fully  networked  forces  engaged 
in  a  convoy  mission  in  an  urban  environment? 

•  What  are  the  requirements  for  agent-based  simulation  software  to  be 
effectively  used  to  examine  communications  in  urban  terrain? 

D.  FLOW  OF  THESIS 

This  brief  introduction  is  followed  by  a  discussion  of  the  simulation  used, 
scenario  overview,  and  measures  of  effectiveness  (MOEs).  The  next  section  provides  a 
detailed  discussion  of  the  design  of  the  experiment  which  is  applied  to  gain  insights  from 
the  model.  All  of  the  parameter  settings  for  the  data  farming  are  explained  and  justified. 
The  subsequent  section  contains  a  statistical  and  analytical  evaluation  of  the  collected 
data  using  regression  techniques  and  data  mining  tools  (e.g.,  regression  trees)  in  order  to 
extract  profitable  insights.  Finally,  the  findings  are  summarized  and  recommendations 
are  given  on  further  studies  that  will  broaden  our  understanding  of  aspects  of 
communication  in  combat  situations. 
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II.  MODEL  DEVELOPMENT 


This  chapter  deals  with  the  model  development.  After  explaining  the  scenario 
background  and  deriving  the  vignette,  the  simulation  software  MANA  is  briefly 
examined,  including  the  software’s  limitations  and  the  associated  consequences  for  the 
model.  The  next  section  introduces  the  model  terrain  and  time  with  their  relation  to  real 
world  measures.  This  is  followed  by  an  outline  of  the  agents  and  their  main  behaviors. 
The  scenario  script  visualizes  the  sequence  of  events  during  a  run.  The  final  section 
discusses  the  measures  of  effectiveness  for  the  model. 

A.  “BLACK  HAWK  DOWN”  SCENARIO 

As  discussed  in  the  introduction  to  this  thesis,  the  likelihood  of  combat  in  urban 
terrain  is  increasing.  Initial  thinking  about  successful  support  for  ground  forces  during 
city  missions  began  after  the  first  unsuccessful  combat  in  the  post  Cold-war  era. 

Filmed  in  the  movie  ‘Black  Hawk  Down,’  a  failed  raid  on  a  gang  leader 
stronghold  was  conducted  by  US  forces  on  October  3,  1993  in  Mogadishu.  US  Rangers 
were  attempting  to  capture  two  high-ranking  lieutenants  serving  Farrah  Aidid,  the  most 
dangerous  warlord  in  Somalia.  The  troops,  sent  in  by  helicopter,  were  to  be  picked  up  by 
a  convoy  and  brought  back  to  the  base.  The  two  targets  were  arrested  successfully. 
However,  the  Somali  militia  reacted  quickly  to  the  attacking  US  soldiers  and  gathered 
behind  the  cover  of  weapons  in  the  streets.  The  militia  then  built  blockades  with  burning 
tires  and  cars.  For  the  US  forces,  this  was  a  new,  unexpected  development  in  comparison 
to  previous  raids. 

During  the  campaign,  two  Black  Hawk  helicopters  were  shot  down  and  the  pilots 
were  surrounded  by  militia  and  a  crowd  of  aggressive  civilians.  The  convoy  was  ordered 
to  retrieve  all  soldiers  from  the  first  crash  site.  The  situation  became  confused,  however, 
because  the  command  cell  in  a  hovering  helicopter  above  the  city  could  not  lead  the 
convoy  through  the  streets  in  avoidance  of  blockades  and  hostile  fire.  The  convoy 
ignored  the  last  order  and  focused  instead  on  finding  a  route  back  to  the  base. 
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Thousands  of  Somalis  took  to  the  streets  and  fought  against  the  outnumbered  US 
Rangers  who  were  left  in  the  city.  The  US  military  launched  a  rescue  campaign 
involving  Pakistani  tanks  and  APCs.  The  rescuers  liberated  all  of  the  soldiers — whether 
dead  or  alive — although  one  pilot  was  held  as  hostage  by  Farrah  Aidid  and  released  after 
several  weeks  in  Somali  captivity.  On  this  day,  18  American  soldiers  died  and  73  were 
injured. 

The  vignette  examined  in  this  thesis  research  is  only  one  part  of  the  ‘Black  Hawk 
Down’  scenario  and  is  translated  into  a  current  Gennan  background.  Several  aspects  of 
this  scenario,  which  is  commonly  used  for  peace-enforcement  situations,  were  examined 
in  previous  studies.  In  order  to  find  insights  about  communication  aspects  in  urban 
environments,  this  study  focuses  on  the  convoy’s  attempt  to  make  its  way  back  to  base. 
Convoy  missions  are  an  everyday  business  for  German  troops  in  Afghanistan. 

1.  Vignette  Description 

The  vignette  is  translated  to  a  newer  world.  German  ground  forces  in  Mazar-E- 
Sharif,  Afghanistan  attack  a  gang  leader  stronghold  and  arrest  the  suspects.  The  forces 
are  picked  up  by  the  convoy,  which  then  must  make  its  way  through  the  street  channels 
and  back  into  the  secure  area  where  the  base  is  located.  The  forces  are  equipped  with 
FAUST,  which  has  automated  data  updates  from  a  sensor  grid  consisting  of  five  hovering 
UAVs  (e.g.,  FANCOPTER). 

The  well-prepared  militia  forces  react  quickly  to  the  attack  and  build  up  blockades 
with  different  time  delays  to  slow  the  convoy’s  movement  and  lead  it  into  an  ambush. 
These  blockades  appear  in  three  waves  with  random  compositions. 

B.  SIMULATION  SOFTWARE  AND  ITS  LIMITATIONS 

1,  MANA 

The  software  package  used  for  implementation  is  MANA  Version  3.2.1.  Map- 
Aware-Non-Uniform  Automata  (MANA)  was  developed  by  the  New  Zealand’s  Defense 
Technology  Agency.  In  the  user’s  manual,9  the  developers  state  that  they  developed  the 

9  MANA  Version  3.0  Users  Manual,  D.  Galligan,  M.  Anderson,  M.  Lauren,  July  2004. 
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new  software  because  their  existing  agent-based  simulation  produced  outputs  which 
differed  radically  from  combat  model’s  results  like  CAEn  and  Janus.  The  intention  was 
not  to  improve  a  cellular  automaton  model,  but  to  create  a  new  model  with  new  features. 

An  agent-based  combat  model  (ABM)  contains  entities  that  are  controlled  by 
decision-making  algorithms. 10  The  agents  react  and  interact  with  their  surroundings 
(e.g.,  terrain,  friends  and  enemies).  MANA  incorporates  several  new  elements.  For 
instance,  the  squad  possesses  two  common  situational  awareness  maps.  These  ‘memory 
maps’  11  store  the  organic  information  12  and  inorganic  information  gained  by 
communications.  The  agent  can  react  differently  to  the  kind  of  information  it  receives. 
The  agent  can  also  change  its  personality  by  an  event-driven  trigger  state  change.  Agent- 
based  models  are  usually  time-stepped  models,  but  as  a  part  of  the  interactions  with  their 
surroundings,  MANA  agents  can  change  their  trigger  states  and  behaviors.  The  duration 
in  a  state  is  variable. 

Agent-based  models  are  intended  to  provide  a  straightforward  way  to  create  a 
scenario  in  order  to  examine  a  special  situation.  In  comparison  to  a  highly-detailed 
combat  model,  the  run-time  is  relatively  short.  Therefore,  multiple  runs  over  a  range  of 
input  values  are  feasible.  An  efficient  design  of  experiment  combined  with  statistical 
analysis  provides  numerous  insights  and  sometimes  surprising  results.  Due  to  the  agent’s 
simple  set  of  rules,  however,  it  is  possible  that  realization  of  ‘correct’  behavior  may  be 
more  difficult  to  achieve. 

2.  Limitations  and  Workarounds 

The  developers  of  MANA  strongly  emphasize  that  the  simulation  software  is  not 
free  of  limitations.  Two  major  limitations  have  become  obvious:  the  greedy  movement 
algorithm  and  the  implementation  of  the  communication  features.  These  problems  and 
their  respective  solutions  are  discussed  in  this  section.  MANA  offers  a  number  of 


10  MANA  Version  3.0  Users  Manual,  D.  Galligan,  M.  Anderson,  M.  Lauren,  July  2004. 

1 1  MANA  Version  3.0  Users  Manual,  D.  Galligan,  M.  Anderson,  M.  Lauren,  July  2004. 

12  Contacts  detected  by  the  agents  themselves  or  by  another  squad  member  are  stored  as  organic 
information  on  the  organic  situational  awareness  map.  On  the  other  hand,  contact  data  that  the  agents 
receive  by  a  communication  link  is  considered  inorganic  information,  and  is  stored  on  the  inorganic 
situational  awareness  map. 
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possibilities  to  circumvent  the  limitations  by  creative  modeling.  Varying  parameters  and 
the  use  of  different  trigger  states  can  often  help  to  overcome  the  software’s  weaknesses. 

a.  The  Greedy  Movement  Algorithm 

As  mentioned  in  the  previous  section,  the  MANA  movement  algorithm 
operates  on  the  agent’s  desires  and  the  terrain.  In  order  to  calculate  the  penalty  function, 
the  software  takes  into  account  only  the  pixels  surrounding  the  agent.  The  penalty  is 
updated  every  time  step.  Projections  into  further  steps  are  not  possible  in  the  software 
version  used  for  this  thesis.  The  agent  makes  its  decision  locally  according  to  its  desires. 
This  differs  substantially  from  human  decision  making.  In  a  real-life  situation,  the 
convoy  leader  gathers  all  available  information  and  processes  it.  His  time  horizon  is 
wider,  and  he  bases  his  decision  on  the  given  situational  awareness  and  projections  into 
the  future. 

In  order  to  achieve  similar  agent  behavior,  two  kinds  of  supporter  agents 
have  been  created.  The  missing  knowledge  about  the  map  is  replaced  by  a  grid  of  friend 
agents  that  send  their  positions  constantly  and  reliably  and  act  as  a  warning  system  when 
a  street  is  blocked  or  there  is  no  continuous  route  after  using  a  street.  The  logic  for 
closing  streets  is  accomplished  by  ‘logic  helpers,’  which  use  reaction  chains  in  special 
situations.  For  example,  when  the  ‘logic  helpers’  at  a  four-way  crossroad  notice  that 
three  of  the  streets  are  blocked,  then  they  will  block  the  fourth  street.  Thus,  these  agents 
help  the  convoy  avoid  an  ambush. 

b.  The  Communication  Features 

The  communication  links  implemented  in  MANA  do  not  mirror  the  reality 
of  current  communication  in  combat.  The  data  transmission  between  two  squads  is  a 
continuous  flow  of  information.  The  sender  transmits  its  organic  situational  awareness 
map  every  time  step.  A  message  containing  the  enemy  type  and  its  location  eventually 
reaches  the  recipient  with  high  probability  even  if  the  reliability  for  the  communication 
link  is  low.  The  high  number  of  trials  makes  this  possible. 

It  is  difficult  to  interpret  the  capacity  and  reliability  of  the  communication 
factors  in  MANA  because  of  the  fact  that,  in  reality,  messages  are  sent  in  packages.  In 
other  words,  the  package  may  or  may  not  reach  the  other  squad.  In  MANA,  each  contact 
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is  put  in  a  single  message  and  sent  with  the  pre-set  reliability  level.  A  partial  transfer  is 
possible  and  can  produce  absurd  situations.  This  means  that  the  convoy  can  have 
knowledge  about  an  ambush  without  seeing  all  associated  blockades.  Unfortunately, 
there  is  no  perfect  solution  to  this  problem,  a  drawback  which  will  be  addressed  in  the 
results  discussion  of  Chapter  IV. 

3.  Aggregation 

Aggregation  is  used  to  reduce  the  complexity  of  the  model.  Agent-based  models 
by  themselves  are  based  on  simplicity.  Therefore,  the  convoy  is  modeled  on  the 
movements  of  only  one  vehicle.  The  intra-convoy  behavior  (movement  in  line  and  intra¬ 
convoy  communications)  is  outside  the  focus  of  this  research  and  is  excluded. 

The  same  approach  and  argumentation  is  used  for  the  blockades.  As  a  military 
rule,  blockades  are  observed,  and  the  blockade  is  merged  with  the  militiamen.  That  is, 
the  blockade  itself  is  armed. 

In  modern  communication  architectures,  the  sensors  are  linked  to  a  Command  & 
Control  Center  (CCC)  that  distributes  the  processed  data  back  to  the  deployed  forces. 
Because  MANA  does  not  allow  the  user  to  disconnect  or  connect  a  communication  link 
by  changing  the  trigger  state,  it  is  nearly  impossible  to  explicitly  model  the  CCC. 
However,  the  necessity  of  modeling  such  an  element  is  questionable.  The  physical 
parameters  like  latency,  reliability  and  duration  between  updating  can  be  viewed  as 
network  parameters.  The  latencies  in  the  sequential  network  add  up,  while  the  resulting 
reliability  is  the  product  of  the  reliabilities  along  the  path.  The  updating  between  sensors 
and  the  CCC  on  one  side  and  the  CCC  and  the  ground  forces  should  be  synchronized. 
Therefore,  the  duration  between  updates  is  a  measure  for  the  update  frequency  of  the  total 
network. 

C.  TERRAIN  AND  TIME 

The  German  Army  has  forces  deployed  in  the  Northern  Province  of  Afghanistan. 
The  German  Headquarters  is  located  in  Mazar-E-Sharif.  For  the  purpose  of  this  scenario, 
one  square  kilometer  out  of  the  Army  city  map  was  scanned.  Microsoft  Paint  was  used  to 
draw  the  contours  and  color  the  map. 
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In  the  scenario,  only  three  terrain  colors  are  used:  black  (streets),  grey  (buildings) 
and  white  (open  terrain).  The  movement  parameters  are  adjusted  to  these  colors. 

The  scenario  terrain  is  not  a  100  percent  copy  of  the  map.  In  order  to  gain  more 
realistic  convoy  behavior,  the  streets  have  uneven  width  and  the  comers  are  softened. 
The  amount  of  open  terrain  is  reduced  to  avoid  unwanted  interactions.  The  areas  around 
the  cemetery  and  refugee  camp  are  totally  remodeled.  The  differences  can  be  observed  in 
Figure  6,  which  shows  the  scanned  map  and  its  ‘translation’  for  the  scenario. 


Figure  6.  The  original  map  of  Mazar-E-Sharif  and  the  MANA  terrain  map. 


The  resolution  employed  in  MANA  is  1000x1000  pixels.  To  ensure  an  easy 
calculation,  one  pixel  represents  one  meter  in  the  real  map  measure. 

The  scale  for  one  time  step  is  designed  as  follows.  In  a  peaceful  environment,  the 
convoy’s  route  stretches  about  1.5  km  long;  it  takes  about  1500  time  steps  to  reach  the 
final  destination.  Operating  on  the  assumption  that  a  convoy  runs  through  a  town  with  an 
average  speed  of  30  km/h,  1500  time  steps  in  the  scenario  are  equal  to  3  minutes  (180 
seconds).  One  time  step  is  thus  about  0.12  seconds. 
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D.  AGENTS  ON  THE  BATTLEFIELD 

This  subsection  describes  the  main  characteristics  of  agents  within  the  simulation. 
Furthermore,  the  section  emphasizes  the  role  of  workarounds. 

1.  Map  Grid 

As  mentioned  in  the  previous  section,  the  convoy  in  the  simulation  does  not  have 
a  map.  It  reacts  only  to  the  local  terrain  and  the  desires  it  possesses  according  to  its  pre¬ 
defined  personality.  Therefore,  friend  agents  have  been  created  to  support  the  convoy  in 
its  course  through  the  streets  of  the  city.  As  dictated  by  the  real  street  network,  the  agents 
are  subdivided  into  three  groups  whose  number  depends  on  the  length  of  the  street  for 
which  they  are  responsible.  They  are  located  at  the  beginning  and  end  of  various  streets. 
Depending  on  the  street  length,  an  agent  has  a  sensor  range  of  30,  60  or  140  meters  to 
guarantee  that  the  agent  can  detect  something  only  in  its  street.  By  default,  the  map  grid 
is  completely  concealed  in  order  to  avoid  interactions.  The  map  grid  has  a  100  percent 
reliable  and  continuous  communication  link  to  the  convoy.  The  locations  of  all  squad 
members  are  transmitted.  This  creates  a  kind  of  street  network  on  the  convoy’s  inorganic 
map.  The  high  reliability  appears  as  the  digital  map  on  a  monitor  mounted  in  front  of  the 
convoy  leader.  The  behavior  of  each  agent  on  the  digital  map  is  characterized  by  three 
functions.  Each  case  is  shown  in  Figure  7. 


Figure  7. 


The  three  possible  states  of  a  map  grid  agent.  (Best  viewed  in  color) 
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•  If  the  street  is  free  and  usable  for  the  convoy,  these  agents  have  the  same 
allegiance  as  the  convoy. 

•  If  an  agent  detects  a  blockade  red  Cross  in  its  street  (threat  level  2),  it 
changes  the  allegiance  and  becomes  an  unconcealed  enemy  with  threat 
level  1.  This  action  is  identical  to  leaving  the  squad;  therefore,  the  agent’s 
location  is  no  longer  transmitted  to  the  convoy. 

•  If  an  agent  detects  a  ‘logic  helper’  in  its  street  (yellow  star  enemy  threat 
3),  it  becomes  an  unconcealed  neutral.  As  in  the  case  of  a  red  Cross,  it 
leaves  the  squad  and  vanishes  from  the  inorganic  map  of  the  convoy. 

The  convoy’s  reaction  to  the  different  states  of  map  grid  agents  will  be  described  in  the 

section  on  convoy  behavior. 

2.  Blockades 

The  scenario  consists  of  three  blockade  squads  (A,  B  and  C)  consisting  of  varying 
numbers  of  agents  (12,  15  and  18).  Blockade  A  becomes  active  when  the  convoy  starts 
its  movement.  Blockade  B  is  hidden  for  215  time  steps  (26  seconds)  and  appears  with  its 
resulting  ‘logic  helpers.’  Finally,  the  last  wave  of  blocked  streets,  blockade  C,  is 
activated  after  another  545  time  steps  (65  sec). 

The  scenario  setting  is  organized  in  the  first  phase  of  the  simulation.  The 
blockade-agent  goes  through  a  sequence  of  states,  as  illustrated  in  Figure  8.  During  the 
RUN  START  state,  the  agent  remains  totally  concealed;  after  a  fixed  duration,  the  agent 
transitions  into  SPARE  1  STATE,  where  it  becomes  visible  and  vulnerable  to  attacks  by 
super-agents.  If  the  agent  is  killed,  it  disappears;  otherwise,  it  steps  into  SPARE  2 
STATE.  The  now  invulnerable  blockade-agent  rests  in  the  state  for  a  fixed  amount  of 
time  in  order  to  set  up  ‘logic  helper’  correctly.  When  this  preparation  is  complete,  the 
agent  transitions  into  SPARE  3  STATE,  where  it  hibernates,  totally  concealed.  The 
blockade  remains  fixed  up  to  the  point  in  the  scenario  timeline  when  its  blockade  wave  is 
activated.  The  active  phase  starts  with  the  DEFAULT  STATE.  The  durations  may  vary 
depending  on  the  blockade  wave  and  the  number  of  necessary  logical  chains. 
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Figure  8.  Sequence  of  states  for  a  blockade-agent. 

As  mentioned,  the  militia  guarding  the  blockade  is  aggregated  with  the  blockade. 
In  this  scenario,  the  militia  consists  of  five  guards  armed  with  100  rounds  of  ammunition 
each.  In  the  DEFAULT  STATE,  the  blockade  possesses  a  weapon  available  with  500 
rounds.  The  probability  of  hitting  the  target  with  a  single  shot  depends  on  the  distance  to 
the  target  (see  Table  1).  There  are  three  levels  implemented.  The  data  entries  are  based 
on  military  norms.  The  probability  entries  represent  one  shot  per  time  step. 


Table  1. 


Distance  to  target 
[meters] 

Probability 

[percent] 

50 

30 

100 

20 

200 

10 

Probabi 


ity  of  a  successful  shot  depending  on  the  distance  to  the  target. 
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The  convoy  can  open  the  blockade  with  its  weapon.  In  this  case,  the  blockade 
suffers  an  injury  and  changes  its  threat  and  class  level  within  the  state  transitions.  The 
convoy  no  longer  recognizes  the  blockade  as  an  attackable  enemy.  No  weapon  is 
available  to  the  blockade  in  the  states  INJURED  and  REACH  FINAL  WAYPOINT.  In 
the  real  world,  all  blockade  militia  have  been  neutralized  and  the  convoy  has  run  through 
the  burning  tires. 

The  numbers  of  injured  blockades  acts  as  a  measure  of  effectiveness  (MOE)  for 
the  examination  of  the  impact  of  communication.  It  signifies  that  the  convoy  has  been 
forced  to  fight  through  rather  than  maneuver  around  a  blockade. 

3.  ‘Logic  Helper’ 

In  order  to  support  the  convoy’s  decision  concerning  which  route  to  take  through 
the  city,  the  simulation  creates  ‘logic  helpers.’  These  helpers  recognize  an  ambush  and 
close  the  street  leading  into  the  trap.  They  give  the  convoy  a  kind  of  human  logic 
thinking. 


The  helper-agents  appear  in  teams  of  two,  each  of  which  has  contrary  allegiance. 
A  short  example  serves  to  illustrate  how  the  helpers  act  and  react  (Figure  9). 


Figure  9.  Demonstration  of  the  ‘logic  helper’ -agent.  (Best  viewed  in  color) 
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The  initial  situation  at  the  crossroad  (a)  develops  as  follows.  The  enemy  has  set 
up  an  ambush  at  a  four- way-crossroad  by  blocking  three  streets.  The  blockades  (red 
crosses) — enemy  threat  2 — have  been  detected  by  the  map-agents,  which  change  their 
behavior  (yellow  crosses).  The  convoy  thus  recognizes  one  street  as  usable  (green 
crosses). 

The  ‘logic  helpers’  consist  of  two  squads:  the  blue  triangles  (b)  and  the  orange  5- 
point-stars.  The  triangle-agent  notices  the  blockades  and  shifts  into  a  trigger  state  in 
which  it  has  a  perfect  weapon  with  one  round  of  ammunition.  Otherwise,  the  agent 
leaves  the  scenario.  The  agent  opens  fire  only  on  the  nearest  5 -point-star-agent,  which 
appears  in  the  simulation  simultaneously  with  the  triangle-agents.  After  firing,  the 
triangle-agent  disappears. 

The  5 -point-star-agent  can  suffer  a  specified  number  of  hits,  depending  on  the 
street  constellation.  In  this  case  (b),  three  hits  are  necessary  to  kill  the  5 -point-star-agent. 
The  death  of  the  squad  member  results  in  a  trigger  state  transition  (SQUAD  DEATH), 
where  the  surviving  5 -point-star-agent  becomes  an  enemy  threat  level  3.  Map-agents 
recognize  this  change  and  become  neutrals,  thereby  closing  the  previously  unblocked 
street  (c). 

From  the  convoy’s  perspective,  the  final  setting  develops  as  follows.  The  map 
grid  friend  detects  the  ‘logic  helper’  in  its  street  and  becomes  a  neutral.  The  convoy  will 
get  the  infonnation  about  the  neutral  via  communication  link.  The  convoy  will  then 
know  the  blockade  constellation  and  the  neutral’s  location.  The  neutral  does  not  allow 
the  convoy  to  enter  the  street  leading  into  the  ambush. 

There  are  a  variety  of  ‘logic  helpers’  which  use  principles  similar  to  those 
outlined  above.  The  trigger  state  transition  of  triangle-agents  can  also  be  activated  by  a 
neutral  or  an  enemy  threat  3.  On  the  other  hand,  the  surviving  5 -point-star-agents  can 
mutate  to  neutrals  and  close  the  road  directly.  When  the  final  setting  is  complete,  the 
helpers,  like  the  blockades  B  and  C,  hibernate  until  the  associated  blockade  wave  enters 
the  scenario. 
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4.  Sensor  Grid 

MANA  structures  communication  as  a  continuous  flow  of  infonnation  along  the 
link.  At  each  time  step  the  organic  and/or  inorganic  information  is  transmitted  to  the 
recipient.  In  a  modem  digital  data  network,  information  is  packed,  sent  and  frequently 
updated.  A  sensor  enters  an  ‘offline’  mode  regarding  the  communication  link  between 
updates  of  the  information. 

In  order  to  achieve  this  behavior,  MANA  implements  a  perfect  cookie-cutter 
sensor  that  alternates  between  two  states  during  the  main  phase  of  the  simulation.  The 
use  of  a  sensor  that  would  frequently  misclassify  contacts  would  lead  to  chaotic  situations 
within  the  scenario.  The  logical  chaining  relies  on  accurate  information  about  detected 
agents,  and  therefore  requires  the  perfect  sensor.  Due  to  the  fact  that  MANA  does  not 
have  a  3 -dimensional  picture,  however,  the  hovering  UAV  is  placed  on  the  ground  and 
can  scan  through  walls.  In  actual  combat  situations,  the  UAV  altitude  would  be  chosen 
in  order  to  have  maximum  view  of  the  ground. 

The  sensor-agent’s  behavior  is  visualized  as  a  flow  chart  in  Figure  10.  After  the 
setting  phase  (RUN  START  STATE),  the  sensor-agent  scans  the  environment  within  its 
sensor  range  and  enters  the  information  onto  its  organic  map,  where  it  remains  for  only 
one  time  step.  This  data  is  sent  to  the  convoy’s  inorganic  map.  There  is  only  one  chance 
to  put  the  data  in  the  communication  queue.  In  the  next  step  (SPARE  2  STATE),  the 
sensor  grid  goes  blind  by  setting  its  sensor  range  to  zero.  The  agent  stays  in  this  mode 
for  a  fixed  duration  before  it  automatically  falls  back  into  SPARE  1  STATE  and  receives 
the  sensor  capability. 


18 


RUN  START 

No  sensor  capability 
during  scenario  setting 

Duration:  100 

Setting  Phase 

r 

Simulation  Phase 

SPARE  1 

Sensor  capability 

Duration:  1 

Tasks: 

Updating  organic  map 

Passing  organic  map  to  the  convoy 

k 

1 

f 

SPARE  2 

No  sensor  capability 

Duration:  variable 

Figure  10.  The  sensor  behavior. 

The  sensor  grid  is  built  by  five  sensors  possessing  a  direct  link  to  the  convoy. 
The  communication  parameters — latency,  reliability,  capacity,  the  sensor  range  and  the 
pause  between  updates  (duration  in  SPARE  2) — vary  as  factors  in  the  simulation.  The 
sensors  remain  in  fixed  positions  during  the  simulation.  Normally,  a  sensor  grid  is  a  self- 
maintaining  system  designed  to  optimize  coverage  of  the  area. 

5.  Super-Agents 

The  scenario  contains  three  squads  of  super-agents  associated  with  one  blockade 
wave.  These  squads  guarantee  a  random  setting  of  the  blockades  within  the  wave. 
Different  intensity  levels  are  created  by  varying  the  number  of  super-agents  per  squad. 
The  number  of  blockades  in  a  specific  wave  thus  becomes  a  controllable  enemy  factor. 

A  super-agent  has  a  perfect  cookie-cutter  sensor  and  a  perfect  weapon  with  one 
round  of  ammunition.  The  agent  reacts  to  the  appearance  of  its  associated  blockade  wave 
(synchronized  by  RUN  START  STATE),  chooses  its  target  randomly  and  kills  it. 
MANA  excludes  the  possibility  that  squad  members  may  shoot  at  the  same  target 
simultaneously.  After  firing,  the  agent  falls  back  into  AMMO  OUT  STATE,  after  which 
it  disappears  from  the  scenario  map. 
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6.  Convoy 

The  convoy  is  simulated  by  one  agent.  The  aggregation,  as  mentioned  before, 
reduces  the  complexity  in  order  to  model  the  correct  convoy  behavior  of  several  vehicles 
and  their  intra-squad  communications.  The  convoy  is  located  at  its  starting  point  during 
the  setting  phase  of  the  scenario  but  does  not  move.  Fuel  status  is  used  as  a  surrogate  for 
urgency  in  returning  to  base,  since  actual  fuel  requirements  are  not  used  in  the  model. 
The  convoy  transitions  automatically  into  the  REFUEL  BY  FRIEND  state.  After  2200 
time  steps  (real  world:  about  4'A  min)  it  jumps  into  the  FUEL  OUT  state.  This  behavior 
change  results  in  a  desire  to  reach  the  base  as  quickly  as  possible.  Figure  1 1 
demonstrates  this  organic  behavior —  information  gathered  by  the  convoy’s  visual 
sensor — by  showing  the  differences  between  the  two  trigger  states. 


Figure  11.  The  convoy’s  organic  setting  in  REFUEL  BY  FRIEND  (left)  and  FUEL 
OUT  (right). 

In  the  beginning  of  the  simulation  phase,  the  convoy  attempts  to  avoid  the 
blockades  that  it  has  visually  detected.  The  map-agents  (Enemy  Threat  1)  strongly  repel 
the  convoy  when  they  are  in  close  range.  The  blockades  themselves  also  have  a  deterrent 
character  within  the  convoy’s  sensor  range  (400  meters).  In  this  scenario,  the  agent 
considers  only  one  waypoint  to  be  its  final  destination — the  base.  The  NEXT 
WAYPOINT  draws  the  agent. 
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When  the  convoy  shifts  into  FUEL  OUT,  the  final  destination  becomes  a  stronger 
focal  point  while  the  blocked  streets  deterrence  parameters  are  reduced.  The  convoy  will 
now  run  through  the  blockades  in  order  to  reach  the  secure  area.  In  both  states,  the 
convoy  has  a  ‘short-distance- weapon’  to  open  a  blockade.  The  agent  can  use  the  weapon 
in  a  radius  of  25  meters  around  a  blockade,  and  will  hit  the  barrier  with  certainty. 


Inoi panic  SA: 


Min  Add,  Max.  Inf, 

Enemy  Threat  1  0  _rJ  _  _D  [50  [200 

Enemy  Threat  2  45  LiJ  _ I  ID  [75  [300  jj] 

Enemy  Threat  3  0  CD  Id  CD  |o  1 10000  *[ 

Friends  55  CD  _l  _D  |o  J\  [200  tj 

Neutrals  -70  GCTCD  CD  |<>  [35 

Unknowns  0  CD  CD  CD  1 10000 


Inoipanic  SA: 

Enemy  Threat  I 
Enemy  Threat  2 
Enemy  Tlueat  3 
Friends 
Neutials 
Unknowns 


Min  Add.  Max.  Inf. 

-15  CD  _l  _iJ  |o  :[|300  t\ 

-15  ld  _i  _d|o  [300  : 
0  LD  _l  _d|o  z\  1 10000 

40  a  cttd|o  I}|2oo 

-15  LD  _l  CD  |o  U|25  U 
0  LD  _ I  _D  |o  II  [10000  t] 


Figure  12.  The  convoy’s  inorganic  settings  parameters  in  the  REFUEL  BY  FRIEND 
(left)  and  FUEL  OUT  (right). 


Similar  to  the  organic  parameters,  the  inorganic  parameters  change  during  the 
transition.  At  the  start,  the  convoy  avoids  streets  blocked  by  ‘logic  helpers’  because  of 
the  repellant  nature  of  neutrals.  In  addition,  the  convoy  uses  its  inorganic  information  to 
stay  away  from  blockades.  The  map  agents  draw  the  convoy  toward  usable  streets.  In 
the  FUEL  OUT  state,  the  levels  for  blocked  streets  are  set  low  (-15)  and  the  friend 
attraction  is  reduced  in  comparison  to  the  previous  trigger  state  to  support  the  organic 
desire  to  reach  the  final  destination. 


E.  SCENARIO  SCRIPT 

The  script  of  the  scenario  is  visualized  in  the  following  figure.  The  figure 
demonstrates  the  subdivision — setting  and  simulation  phase — and  the  associated  time 
steps  between  the  events.  One  purpose  of  separating  the  setting  and  simulation  phase  lies 
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in  the  comparability.  Using  the  same  seed13,  setting  identical  levels  for  the  blockades 
and  varying  only  the  communication  factors  result  in  the  same  enemy  set  up.  This  makes 
a  pair-wise  comparison  of  runs  possible. 

The  events  are  summarized  for  the  different  agents  in  Figure  13. 


Figure  13.  The  scenario  script. 


13  From  the  MANA  Manual  p.  14:  “Seed:  This  number  is  used  to  generate  all  the  random  numbers 
used  in  a  run  (start  position,  hit  probabilities  etc.).”  It  was  observed  that  small  changes  in  the  terrain  file 
and  in  the  numbers  of  agents  placed  on  the  battlefield  lead  to  different  blockade  settings,  even  using  the 
same  seed. 
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F.  MEASURE  OF  EFFECTIVENESS 

MANA  generates  an  EXCEL  file  that  includes  several  fields,  such  as  ‘Reach 
Final  Waypoint,’  run  time,  number  of  casualties  in  each  squad  and  number  of  injured 
members  within  a  squad.  This  file  contains  one  row  per  run.  The  software  package  also 
delivers  one  output  file  per  run  which  stores  all  agents’  end  state  data.  This  file  shows 
the  state  in  which  the  agent  ended  its  final  location  and  the  number  of  hits. 

The  MOEs  for  the  simulation  are  derived  and  extracted  from  this  available  data 
collection.  These  MOEs  are  directly  related  to  the  research  focus  and  will  be  defined 
precisely.  They  are  mutually  exclusive  in  order  to  avoid  overlapping. 

•  Communications  influence  the  convoy’s  route  through  the  street  channels. 
Therefore,  the  mission  length  is  deduced  by  subtracting  the  setting  phase 
from  the  run  time.  The  linear  transformation  is  done  in  the  data  post¬ 
processing. 

•  The  precision  of  the  information  transmitted  via  the  communication  link 
leads  the  convoy  to  pursue  different  paths  through  the  city,  resulting  in 
different  numbers  of  broken  blockades.  This  number  is  also  a  MOE,  and 
is  calculated  by  adding  the  number  of  injured  members  within  the 
blockade  squads. 

•  Communications  affect  the  number  of  hits  the  convoy  suffers.  A  Ruby14 
script  is  used  to  read  the  data  out  of  the  agent  end  state  files.  The  routine 
opens  the  file  by  the  predefined  filename  and  reads  out  the  identification 
(run  and  excursion),  the  needed  information  final  location  and  the  number 
of  hits. 

•  In  the  case  that  the  convoy  does  not  reach  its  final  destination,  its  last 
position  in  the  scenario  leads  to  conclusions  about  the  cause  of  its  failure. 

An  explanation  of  which  MOEs  are  the  most  reliable  indicators  is  made  in 
Chapter  IV,  after  the  analysis  of  the  collected  data. 


14  Ruby  is  an  open  source  programming  language  which  provides  excellent  scripting  and  data 
manipulation  capabilities.  Available  at  http://www.ruby-lang.org/en. 
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III.  DESIGN  OF  EXPERIMENT 


Combined  with  an  efficient  Design  of  Experiment  (DOE),  the  MANA  simulation 
tool  provides  a  good  seed  for  the  data  fanning  and  promises  a  successful  harvest.  This 
chapter  focuses  on  the  DOE  used  to  verify  the  model  and  gain  insights  about 
communication  aspects  in  urban  terrain.  First,  the  Nearly  Orthogonal  Latin  Hypercube 
(NOLH),  the  main  DOE  technique  used  in  this  thesis,  is  introduced  together  with  its 
strengths  and  weaknesses.  Factors  for  the  scenario  are  divided  into  two  groups, 
controllable  and  uncontrollable.  The  design  used  for  each  group  is  explained.  In  the  last 
step,  the  designs  for  assessing  the  model  quality  and  examining  communication  aspects 
are  derived. 

A.  THEORY  OF  NEARLY  ORTHOGONAL  LATIN  HYPERCUBES 

Insight  occurs  most  efficiently  in  experiments  that  vary  several  factors 
simultaneously.  One  possibility  is  a  full  factorial  design.  Such  a  design  contains  all 
combinations  between  all  factor  levels.  Despite  the  ready  availability  of  computing 
power  today,  the  number  of  runs  increases  exponentially  upon  the  addition  of  factors  and 
levels.  In  order  to  examine  4  factors  with  10  levels  each,  10,000  (=  104)  design  points 
would  be  necessary.  This  must  further  be  multiplied  by  the  number  of  replications  for 
each  design  point.  The  total  number  of  runs  quickly  reaches  millions  or  more  when 
several  factors  are  explored.  These  kinds  of  gridded  designs  are  thus  useful  only  for  a 
small  number  of  factors  and  levels. 

Lieutenant  Colonel  Thomas  Cioppa,  an  NPS  PhD  graduate,  researched  in  the  area 
of  experimental  design  and  found  a  ‘smart’  solution  to  reduce  the  number  of  design 
points  while  still  covering  the  factor  space  efficiently.  In  his  dissertation,15  he  examined 
the  use  of  Nearly  Orthogonal  Latin  Hypercubes.  The  main  characteristics  for  a  NOLH 
design  and  the  resulting  consequences  are  briefly  summarized: 

15  Lt.Col  Tomas  M.  Cioppa,  “Efficient  Nearly  Orthogonal  and  Space-Filling  Experimental  Designs 
for  Eligh-Dimensional  Complex  Models,”  PhD  dissertation,  Operations  Research  department.  Naval 
Postgraduate  School,  Monterey,  2002. 
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•  The  input  factors  are  nearly  uncorrelated.  Therefore,  the  design  has 
properties  that  accommodate  statistical  analysis.  Specifically, 
multicollinearity  is  not  a  concern  when  using  linear  regression. 

•  The  space-filling  properties  of  the  design  points  sample  across  the  input 
space. 

•  The  reduction  of  design  points  when  compared  to  a  full  factorial  design, 
does  not  substantially  decrease  the  capability  to  analyze  many  effects  and 
interactions. 

A  NOLH  design  opens  the  opportunity  to  simultaneously  examine  a  large  number 
of  input  factors.  Although  this  thesis  does  not  focus  on  the  theory  and  mathematical 
background  for  NOLHs,  it  is  a  useful  tool  to  explore  a  variety  of  input  factors  within  a 
reasonably  sized  experiment. 

A  small  example,  see  Figure  14,  demonstrates  the  differences  between  a  2k- 
factorial  design  and  NOLH-design  involving  four  continuous  factors  with  a  range  of  1  to 
10.  A  low-high  (2  level)  design  is  compared  to  a  NOLH  design16  with  17  design  points. 
The  number  of  design  points  are  nearly  identical  with  16  (24)  and  17,  but  the  spread 
across  the  4-dimensional  input  space  is  different,  as  evident  in  the  pair-wise  scatterplots. 
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Figure  14.  Comparison  between  2k-factorial  design  and  NOLH  with  4  factors. 


16  This  design  uses  the  spreadsheet  available  at  http:/harvest. nps.edu/  <Software  Download?-. 
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The  2k-factorial  design  covers  only  the  comers,  while  the  NOLH  fills  the  space 
without  touching  the  extreme  points  in  the  comers.  Both  designs  are  orthogonal,  as 
shown  in  the  correlation  matrices  (Figure  15). 


2k-factorial  Design 


Collections 


Factor  1 

Factor  2 

Factor  3 

Factor  4 

Factor  1 

1 ,0000 

0,0000 

0,0000 

0,0000 

Factor  2 

0,0000 

1 ,0000 

0,0000 

0,0000 

Factor  3 

0,0000 

0,0000 

1 ,0000 

0,0000 

Factor  4 

0,0000 

0,0000 

0,0000 

1 ,0000 

NOLH  Design 


Correlations 


Factor  1 

Factor  2 

Factor  3 

Factor  4 

Factor  1 

1 ,0000 

0,0000 

0,0000 

0,0000 

Factor  2 

0,0000 

1 ,0000 

0,0000 

0,0000 

Factor  3 

0,0000 

0,0000 

1 ,0000 

0,0000 

Factor  4 

0,0000 

0,0000 

0,0000 

1 ,0000 

Figure  15.  Comparison  of  the  correlation  matrices  of  a  2k- factorial  design  and  NOLH 
design. 

The  only  noticeable  disadvantage  of  a  NOLH  design  is  the  lack  of  coverage  at  the 
edges.  Extreme  low  and  high  settings  for  input  factor  combination  are  not  achievable 
with  this  kind  of  design.  The  next  section,  which  deals  with  uncontrollable  factors, 
shows  a  solution  to  cover  the  edges. 


B.  NOISE  FACTORS 

The  introduction  of  uncontrollable  factors,  also  called  noise  factors,  is  necessary 
to  examine  the  impact  of  communication  aspects  on  a  broader  base.  The  enemy’s 
behavior  and  capabilities  affect  outcomes,  but  are  outside  of  blue  control.  Modeling 
them  explicitly,  but  classifying  them  as  noise  factors,  allows  the  experimenter  to  test  the 
robustness  of  proposed  solutions.  All  controllable  combinations  of  the  communications 
aspects  are  tested  in  several  scenarios  with  different  noise  factor  settings,  such  as  the 
density  of  blockades. 

The  number  of  blockades  per  wave  is  directly  controlled  by  the  number  of  super¬ 
agents  associated  with  the  wave.  In  a  first  approach,  a  NOLH  design  for  the  noise  factors 
is  examined.  The  use  of  the  NOLH  Excel  spreadsheet  leads  to  a  DOE  with  17  design 
points.  Table  2  presents  the  chosen  ranges.  A  large  range  has  been  chosen  to  stress  the 
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communication  factors.  Although  it  is  clear  that  15  blockades,  the  upper  bound,  is  an 
unrealistically  high  number,  the  range  is  designed  to  also  explore  results  for  a  more 
reasonable  number  of  blockades. 


Number  of  Agents 

Resulting  Blockades  per 
Wave 

Max 

Min 

Min 

Max 

SuperAgent  1 1 

11 

7 

1 

5 

SuperAgent  22 

14 

10 

1 

5 

SuperAgent  33 

17 

13 

1 

5 

Wanted  Total  number  of 
Blockades 

3 

15 

Table  2.  NOLH  settings  for  the  super-agents  and  the  resulting  blockade  outcomes. 


The  actual  resulting  range  for  the  total  number  of  blockades  is  5  to  12.  The 
NOLH-design  does  not  explore  the  values  below  five  and  greater  than  12.  These  ranges 
are  also  important,  however.  Therefore,  eight  new  design  points  are  attached:  one  setting 
for  achieving  15  blockades  and  seven  settings  to  fill  the  space  below  five.  These  seven 
settings  are  derived  by  taking  one  blockade  per  wave.  The  complete  Excel  spreadsheet  is 
contained  in  Appendix  Table  4.  The  distribution  of  the  resulting  blockades  is  shown  in 
Figure  16. 


Total  Blockades 


Figure  16.  The  distribution  of  the  resulting  numbers  of  blockades.  The  marked  parts 
are  the  NOFH-elements. 
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The  additional  design  points  adversely  affect  the  orthogonality  of  the  design.  As 
visualized  in  the  correlation  matrices  in  Figure  17,  the  entries  above  and  below  the  main 
diagonal  are  about  0.5,  which  clearly  differs  from  0.  The  correlation  matrix  for  the  pure 
NOLH  is  given  as  a  comparison,  where  the  absolute  values  of  the  off-diagonal  entries  are 
negligible.  The  differences  in  correlation  occur  when  the  numbers  are  rounded  to 
integers. 


Correlations 

Super  Agentl  1  Super  Agent22  SuperAgent33 
Super  Agemtl  1  1,0000  -0,0805  -0,0085 

Super  Agent22  -0,0805  1,0000  -0,0085 

Super  Agent33  -0,0085  -0,0085  1 ,0000 


Correlations 

Super  Agentl  1  Super  Agent22  SuperAgent33 
Super  Agemtl  1  1,0000  0,4956  0,5271 

Super  Agent22  0,4956  1,0000  0,5271 

Super  Agent33  0,5271  0,5271  1,0000 


Figure  17.  The  correlation  matrix  for  the  pure  NOLH  (left)  and  the  combined  design 
(right). 


The  disadvantage  is  acceptable  because  this  design  is  crossed  with  another 
NOLH-design.  Each  design  point  is  combined  with  each  design  point  of  the  other  design. 
The  correlation  within  one  of  the  designs  does  not  affect  the  result  of  the  Cartesianly 
joined  design.  The  resulting  combinations  are  mutually  orthogonal.  The  correlation 
matrix  for  a  crossed  design  is  detailed  later  in  this  thesis. 


C.  CONTROLLABLE  FACTORS 

The  controllable  factors  in  this  research  are  parameters  that  directly  or  indirectly 
influence  the  blue  force  communication  capabilities  and  are  controllable  by  the  blue 
forces.  This  thesis  explores  seven  factors  of  the  scenario.  These  factors  are  defined  in 
the  following  section.  Attributes  for  the  sensor  grid  are  changed  simultaneously  for  all 
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squads  in  a  lock-stepped  mode.  Instead  of  increasing  the  number  of  factors,  the  lock¬ 
stepping  is  scripted  in  the  XML17  study-file  created  by  a  special  software  tool  called 
Tiller.18 

The  first  six  factors  of  the  scenario  in  question  are  physical  parameters  of  the 
sensor  grid  agents.  The  last  factor  is  a  parameter  directly  affecting  the  convoy.  When 
possible,  the  MANA  definition  found  in  the  User’s  Manual19  is  given  or  an  explanation  is 
provided  as  to  how  this  factor  was  created  in  the  scenario. 

•  Latency:  (MANA  definition)  Number  of  time  steps  taken  for  each 
message  to  reach  the  squad. 

•  Reliability:  (MANA  definition)  Likelihood  that  a  given  message  will  be 
successfully  sent  on  a  link  per  try.  Repeated  attempts  are  made  at 
resending  unsuccessful  messages  until  they  are  successfully 
communicated. 

•  Capacity:  (MANA  definition)  Number  of  messages  that  can  be  sent 
through  the  link  per  time  step.  This  factor  corresponds  to  the  bandwidth  in 
real  world  terms.  Due  to  the  workarounds,  the  number  of  messages  is  not 
linear  to  the  number  of  agents  on  the  battlefield.  This  factor  allows  for 
assessment  of  effects  and  interactions,  but  a  one-to-one  mapping  is  not 
possible. 

•  Pause  duration  between  Updating:  This  factor  describes  the  amount  of 
time  the  sensor  grid  pauses  before  sending  an  update.  MANA  does  not 
provide  a  direct  data  entry;  therefore,  a  workaround  is  created  that 
effectively  blinds  the  sensor  for  the  pause  duration. 

•  Communication  range:  Maximum  range  between  the  centroid  of  the 
transmitting  and  receiving  squads.  The  communication  range  as  a  factor  is 
only  applied  in  the  design  for  assessing  the  model’s  behavior,  and  varies 
in  binary  style  (0  m  vs.  2000  m).  This  factor  is  used  as  surrogate  for 
whether  or  not  a  communication  link  is  available. 


17  The  definition  for  XML  comes  from  www.wikipedia.org:  “The  Extensible  Markup  Language 
(XML)  is  a  W3C-recommended  general-purpose  markup  language  that  supports  a  wide  variety  of 
applications.  XML  languages  or  “dialects”  are  easy  to  design  and  to  process....  Its  primary  purpose  is  to 
facilitate  the  sharing  of  data  across  different  information  systems,  particularly  systems  connected  via  the 
Internet.” 

18  The  Tiller  is  a  software-tool  provided  by  Project  Albert  to  connect  the  scenario  with  the  DOE. 
Based  on  the  MANA  scenario,  the  input  factors  and  their  ranges  are  entered  and  the  kind  of  design  is 
chosen.  The  software  will  automatically  create  a  study  including  the  DOE.  This  folder  contains  the 
scenario  XML-file,  the  terrain  and  the  XML-study  file.  The  study  file  can  be  used  directly  by  the  high 
performance  computer. 

19  MANA  Version  3.0  Users  Manual,  D.  Galligan,  M.  Anderson,  M.  Lauren,  July  2004. 
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•  Sensor  range:  (MANA  definition)  The  radius  in  cells  in  which  an  agent 
can  detect  and  classify  targets. 

•  MaxAge  (Inorganic  threat  Persistence):  (MANA  definition)  The  number 
of  time  steps  that  must  pass  before  a  contact  on  the  squad  inorganic 
situational  awareness  disappears.  MaxAge  is  set  at  1.5  *  Latency.  This 
means  that,  at  the  minimum,  the  contact  remains  on  the  map  for  the  delay 
time  in  the  network;  the  contact  may  also  stay  an  additional  time.  When  a 
contact  is  on  the  map,  it  may  influence  an  agent’s  decision. 

All  seven  controllable  factors  of  the  scenario  are  designed  to  provide  insights 
about  communication  aspects  in  urban  terrain.  Their  input  ranges  are  outlined  in  the 
Table  3. 


MANA  Scenario 

Real  Wor 

Id 

Lower 

Upper 

Lower 

Upper 

Latency 

50 

500 

6  sec 

60  sec 

Reliability  [%] 

10 

100 

10 

100 

Capacity 

300 

500 

Not  directly  convertible 

Pause  between 
Updates 

20 

200 

2.5  sec 

25  sec 

Sensor  Range 

300 

500 

300  m 

500  m 

MaxAge 

Lockstep  function:  1.5  *  Latency 

Table  3.  Input  factor  ranges. 


With  the  exception  of  MaxAge,  all  of  these  factors  are  entered  in  the  spreadsheet 
for  creating  the  NOLH  design.  In  this  case,  a  65  design  point  NOLH  has  been  chosen  to 
achieve  a  dense  spread.  The  map  clearing  frequency  is  added  in  the  spreadsheet  later  as 
an  extra  column. 

D.  DESIGN  FOR  ASSESSING  THE  MODEL  QUALITY 

The  experiment  explained  in  the  following  section  tests  the  model  in  order  to 
explore  the  influence  of  MANA  limitations  on  the  scenario  discussed  in  this  thesis. 

To  gain  insights,  two  different  scenarios  were  established  for  this  design  of 
experiment  (DOE).  The  first  scenario  excludes  the  communication  aspects  between  the 
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sensor  grid  and  the  convoy.  In  other  words,  the  convoy  has  only  visually  collected 
information  available.  On  the  other  hand,  the  convoy  can  rely  on  perfect  information  in 
the  second  scenario.  In  this  case,  perfect  infonnation  is  given  by  a  latency  of  zero  time 
steps,  a  reliability  of  100  percent,  capacity  of  500  messages  per  time  step,  pauses  between 
updates  of  10  time  steps,  and  a  sensor  range  of  350m  for  the  sensor  grid.  . 

The  DOE  is  set  up  by  crossing  the  noise  settings  (25  design  points)  with  a  binary 
communication  range  for  the  sensors  (0m  and  2000m).  The  change  for  the 
communication  range  is  lock-stepped  for  all  five  sensors  in  the  grid.  The  DOE  conducts 
100  replications  for  each  of  the  50  design  points. 

There  are  several  ways  to  eliminate  communication  in  the  scenarios.  This 
experiment  alters  the  physical  settings  of  the  sensor  grid  instead  of  disabling  the  agents 
themselves  in  the  scenario.  A  change  in  the  number  of  agents  placed  on  the  battlefield 
would  result  in  a  different  blockade  setting  for  a  seed  used  in  the  run  with  and  without 
communications.  Therefore,  the  pair-wise  comparability  would  get  lost.  Although  it  is 
also  possible  to  set  the  inorganic  desires  of  the  convoy  to  0,  this  method  requires  several 
changes — specifically,  the  involvement  of  2  trigger  states  and  3  desires. 

A  fixed  set  of  100  seeds20  guarantees  that  each  seed  provides  the  same  blockade 
setting,  given  the  same  number  of  super  agents,  but  independent  from  the  communication 
settings.  Because  of  this  fact,  runs  with  and  without  communication  are  directly 
comparable  in  the  analysis  phase.  The  paired  t-test,  the  most  powerful  tool  for  comparing 
means  of  two  normal  samples,  is  applicable,  as  Chapter  IV  will  make  evident. 


E.  DESIGN  FOR  EXPLORING  COMMUNICATION  ASPECTS 

This  section  explains  the  design  to  capture  the  impact  of  communication  factors. 
As  mentioned,  this  test  employs  a  crossed  design  using  the  controllable  and 
uncontrollable  factors.  Each  design  point  of  the  noise  factors  is  tested  against  each 


20  The  Tiller  software  generates  a  set  of  100  seeds.  The  set  is  applied  to  all  design  points.  This  means 
that  replication  #1  on  design  point  #1  has  the  same  seed  like  replication  #1  on  design  point  #2. 
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design  point  of  the  controllable  factors.  The  broad  approach  ensures  that  the 
experimenter  may  distinguish  between  factors  that  dominate  the  scenario  and  factors  that 
can  be  ignored  in  further  experiments. 

The  crossed  design  consists  of  1625  design  points,  each  with  100  replications. 
The  correlation  matrix  in  Figure  18  shows  that  there  is  no  correlation  between  the  noise 
and  controllable  factors  (see  box).  As  a  result,  useful  properties  for  the  analysis  are 
provided.  This  matrix  also  shows  the  strong  correlation  between  the  factors  Latency  and 
MaxAge,  the  linear  function  of  Latency  (see  oval).  Instead  of  presenting  the  level  of  the 
super  agents,  which  are  varied,  the  number  of  resulting  blockades  is  used  to  calculate  the 
correlation  matrix.  The  number  of  blockades  is  achieved  by  a  linear  transformation  of 
the  number  of  associated  super  agents.  Therefore,  the  correlations  among  the  blockades 
are  identical  to  the  correlations  among  the  super  agents. 


Coiiehitions 


Blockade  A  Blockade  B  Blockade  C  Latency  Reliability  PAUSE  duration  SENSOR  Range  Capacity  MaxAge 


Blockade  A 

1 ,0000 

0,4956 

0,5271 

-0,0000 

-0,0000 

0,0000 

-0,0000 

0,0000 

0,0000 

Blockade  B 

0,4956 

1 ,0000 

0,5271 

0,0000 

0,0000 

0,0000 

0,0000 

0,0000 

0,0000 

Blockade  C 

0,5271 

0,5271 

1 ,0000 

-0,0000 

-0,0000 

-0,0000 

-0,0000 

0,0000 

-0,0000 

Latency 

-0,0000 

0,0000 

-0,0000 

1 ,0000 

-0,0013 

0,0008 

0,0005 

-0,0008 

CT  ’oodiSi 

Reliability 

-0,0000 

0,0000 

-0,0000 

-0,0013 

1 ,0000 

-0,0031 

0,0006 

0,0007 

-0,0013 

PAUSE  duration 

0,0000 

0,0000 

-0,0000 

0,0008 

-0,0031 

1 ,0000 

0,0003 

0,0001 

0,0009 

SENSOR  Range 

-0,0000 

0,0000 

-0,0000 

0,0005 

0,0006 

0,0003 

1 ,0000 

-0,0002 

0,0004 

Capacity 

0,0000 

0,0000 

0,0000 

-0,0008 

0,0007 

0,0001 

-0,0002 

1 ,0000 

-0,0008 

MaxAge 

0,0000 

0,0000 

-0,0000 

1 ,0000 

-0,0013 

0,0009 

0,0004 

-0,0008 

1 ,0000 

Figure  18.  The  correlation  matrix  for  this  communication  aspect  design.  By  crossing 
the  two  designs  Cartesianly,  orthogonality  is  obtained  between 
controllable  and  uncontrollable  factors,  even  when  there  is  correlation 
among  factors  within  each  design. 
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IV.  DATA  ANALYSIS 


This  chapter  focuses  on  data  processing  and  data  analysis.  It  provides 
information  about  the  software  package  that  is  used  and  statistical  tools  that  are  applied  to 
gain  insights  from  the  model.  The  next  section  describes  how  the  raw  data  is  cleaned  up 
and  prepared  for  the  analysis.  In  the  first  analysis  part,  two  important  questions  are 
raised: 

•  How  well  is  the  model  doing? 

•  How  reliable  is  the  model  in  examining  the  focus  of  the  research? 

The  answers  allow  drawing  conclusions  about  the  usefulness  of  modeling  a 
scenario  like  this  in  MANA  by  choosing  an  approach  similar  to  that  of  the  author. 
Finally,  the  main  experiment  is  analyzed  and  the  results  are  presented. 

A.  STATISTICAL  SOFTWARE  PACKAGE  JMP 

To  conduct  an  analysis  on  a  large  data  set  a  professional  statistical  software 
package  is  necessary.  The  author  considered  several  packages  and  decided  to  use  JMP 
The  Statistical  Discovery  Software™.  In  the  manual,  the  software  developers  state  that 
JMP  is  easy  to  learn.  The  author  agrees  on  this  point  and  is  convinced  of  its  user 

friendly  appearance.  Some  advantages  experienced  while  working  with  JMP  are  briefly 

22 

summarized : 

•  GUI  interface  for  all  activities. 

•  Spreadsheet  features  for  manipulating  large  data  sets. 

•  A  broad  range  of  graphical  and  statistical  methods  for  data  analysis. 

•  Capability  of  grouping  data  and  computing  summary  statistics. 

•  Tools  for  printing  and  moving  analysis  results  between  applications. 

JMP  provides  the  entire  tools  necessary  for  the  discovery  tour  through  the  data  sets  in 
this  thesis  research. 


21  JMP  The  Statistical  Discovery  Software,  Introductory  Guide  Version  5. 

22  JMP  The  Statistical  Discovery  Software,  Introductory  Guide  Version  5. 
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B.  RAW  DATA  POSTPROCESSING 

The  raw  data  returned  by  the  Maui  High-Performance  Computer  Center 
(MHPCC)  consists  of  two  types.  One  is  an  EXCEL  file  containing  the  factor  levels  and 
various  squad  outputs,  e.g.,  the  number  of  casualties  or  injuries.  This  file  can  directly 
import  into  JMP  and  be  manipulated  within  the  software  tool.  The  other  data  collected 
are  the  agents’  end  data  for  each  run.  In  order  to  extract  the  main  entries,  a  small  Ruby 
program  (see  Appendix  Figure  36)  was  written  and  applied.  This  tool  reads  in  each  file 
and  puts  the  important  facts,  i.e.,  number  of  hits  and  final  location  of  the  convoy,  in  an 
extra  text  file. 

The  JMP  software  package  has  good  spreadsheet  characteristics,  as  mentioned  in 
the  previous  section.  In  the  next  step  the  columns  of  the  csv  file  and  the  columns  of  the 
text  file  are  joined.  However,  only  a  few  of  the  more  than  hundred  columns  are 
interesting  and  necessary  for  this  research  thesis.  Hence,  the  others  are  deleted. 

Data  entries  are  converted  and  the  following  new  columns  are  added  in  order  to 
achieve  more  readability. 

•  New  column:  The  number  of  super  agents  per  wave  is  translated  into 
blockades  per  wave. 

•  New  column:  The  mission  length  is  calculated  by  subtracting  100  time 
steps  from  the  scenario  run  time  due  to  the  setting  phase  which  was 
explained  in  Chapter  II. 

•  New  columns:  All  time  step  measures  are  converted  into  seconds.  This  is 
done  for  all  factors  and  results. 

As  mentioned  in  Chapter  II,  the  MOEs  are  the  mission  length,  the  number  of  hits,  and  the 
number  of  broken  blockades.  The  last  one  is  the  sum  of  three  columns,  specifically  the 
number  of  injured  blockades.  Also  calculated  is  the  total  number  of  blockades. 

C.  STATISTICAL  METHODS 

After  preparing  the  data  set  the  tools  used  to  gain  insights  are  briefly  introduced. 
The  analyst’s  tool  bag  consists  of  graphical  tools,  parametric  and  non-parametric 
procedures,  multiple  regression,  and  regression  trees. 
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1.  Graphical  Analysis 

A  commander  believes  more  of  what  he  sees  than  what  he  is  told.  Following  this 
common  staff  slogan,  the  data  is  analyzed  with  simple  graphical  tools,  e.g.,  histograms. 
Graphs  and  plots  provide  the  first  impression  about  some  insights  and  ease  the 
interpretation  and  explanation,  especially  to  higher  ranking  officers  without  detailed 
knowledge  about  analysis  techniques. 

2.  Parametric  and  Non-parametric  Procedures 

Parametric  procedures  are  based  on  distributional  assumptions.  The  most  common 
parametric  tests  require  nonnality,  i.e.,  nice  bell-shaped  behavior  in  the  data  or  at  least  for 
the  estimators.  On  the  other  hand,  non-parametric  procedures  are  often  referred  as 
“distribution-free.”  The  data  analysis  in  this  research  uses  the  paired  t-test  as  a  member 
of  the  parametric  procedures  and  the  Shapiro-Wilk  W-test  from  the  non-parametric 
procedures. 

•  The  verification  experiment  delivers  paired  data.  The  paired  t-test  is 
conducted  to  investigate  whether  the  two  samples  could  have  the  same 
mean.  The  test  requires  independent  and  normally  distributed  sample 
means.24  Fortunately,  the  t-test  is  robust  against  non-normally  distributed 
data  for  large  sample  sizes.  The  paired  t-test  focuses  on  means.  The 
Central  Limit  Theorem  states  that  the  sample  mean  is  approximately 
normally  distributed  given  a  sufficiently  large  sample  size.25 

•  The  Shapiro-Wilk  W-test 26  is  a  “distribution-free”  procedure.  The 
Shapiro-Wilk  W-test  examines  the  hypothesis  that  the  sample  comes  from 
a  normal  population. 

3.  Regression  Trees 

A  regression  tree  is  a  Data  Mining27  technique.  “There  is  no  implicit  assumption 
that  the  underlying  relationships  between  the  predictor  variables  and  the  dependent 


23  Jay  L.  Devore,  “Probability  and  Statistics,”  6th  Edition,  2004,  p.  667. 

24  Jay  L.  Devore,  “Probability  and  Statistics,”  6th  Edition,  2004,  p.  381. 

25  Jay  L.  Devore,  “Probability  and  Statistics,”  6th  Edition,  2004,  p.  239. 

26  W.J.  Conover,  “Practical  Nonparametric  Statistics,”  3rd  Edition,  1999,  p.  450.  If  the  test  statistics 
W  “is  close  to  1.0  the  sample  behaves  like  a  normal  population.”  If  W  “is  too  small  and  far  below  1.0,  the 
sample  looks  non-normal.” 

27  David  Hand,  “Principles  of  Data  Mining,”  2001,  p.  2,  Definition:  “Data  mining  is  the  analysis  of 
(often  large)  observational  data  sets  to  find  unsuspected  relationships  and  to  summarize  the  data  in  novel 
ways  that  are  both  understandable  and  useful  to  the  data  owner.” 
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variable  are  linear,  follow  some  specific  non-linear  link  function,  or  that  they  are  even 
monotonic  in  nature.”28  More  advantages  of  this  assumption-free  tool  are  listed  in  the 
JMP  help  as  follows: 

•  “it  is  good  for  exploring  relationships  without  having  a  good  prior  model 

•  it  handles  large  problems  easily,  and 

•  the  results  are  very  interpretable.”29 

The  regression  tree  method  is  applied  to  a  continuous  response  variable  and  input 
factors  which  can  be  continuous  or  categorical.  The  tree  structure  is  built  by  binary 
recursive  partitions.  The  goal  is  to  minimize  the  sum  of  squared  deviations  within  the 
separated  groups.  This  process  increases  the  “purity”  in  the  class.30  The  sequence  of 
factors  involved  in  partitioning  allows  drawing  conclusions  on  their  importance. 
Consecutive  splits  in  a  branch  where  different  factors  are  involved  may  indicate  an 
important  interaction.  Another  combination,  i.e.,  one  factor  showing  up  twice  in  a  row 
may  be  an  indication  of  a  non-linear  effect.  The  analyst  gets  an  idea  on  a  potential  linear 
regression  model.  On  the  other  hand,  factor  levels  are  noticeable  when  a  split  occurs.  In 
some  cases,  when  a  significant  improvement  is  noticed,  this  factor  level  can  be 
interpreted  as  a  threshold. 

4.  Multiple  Linear  Regression 

“Regression  analysis  is  one  of  the  most  widely  used  techniques  for  analyzing 
multifactor  data.”31  In  this  thesis,  multiple  linear  regression  analysis  is  applied.  It  deals 
with  one  continuous  response  and  continuous  and  categorical  input  factors,  called 
predictors  or  regressors.  The  response  is  a  linear  combination  of  coefficients  of  predictors 
and  an  additive  statistical  error.  This  error  is  often  assumed  to  be  uncorrelated  and 
normally  distributed  with  mean  0  and  constant  variance. 


28  StarSoft  Technologies,  www.slalsolt.com/textbook/stcart.htin,  December  2006. 

29  JMP  5.2.1  online-help  Index  “Partition.” 

30  David  Hand,  “Principles  of  Data  Mining,”  2001,  p.  343. 

31  Douglas  C.  Montgomery,”  Introduction  to  Linear  Regression  Analysis,”  3rd  Edition,  2001,  p.  xiii. 
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Regression  models  are  useful  to  describe  data,  estimate  parameters,  predict  and 
estimate  outcomes.  Of  course,  agent-based  models  are  not  built  for  prediction  purposes, 
as  the  author  mentioned  in  Chapter  II.  In  this  thesis,  regression  models  are  used  to  gain 
insights  about  the  important  factors  and  their  interactions. 

D.  ANALYSIS  FOR  ASSESSING  THE  MODEL’S  QUALITY 
1.  Unfinished  Runs 

One  technique  to  consider  on  whether  this  model  is  doing  a  “good”  job  is  a  look 
at  the  runs  and  which  of  them  are  terminated  by  the  time  limit.  Occasionally,  the  convoy 
gets  “stuck”  along  the  way.  This  would  never  happen  in  the  real  world.  Within  the  data 
set  of  5000  runs,  only  in  13  cases  (0.28  %)  did  the  convoy  not  reach  the  final 
destination — that  is,  it  got  stuck  on  its  way.  Figure  19  shows  the  distribution  of  the  last 
location  on  the  map  as  a  function  of  the  communication  range. 


CommRiinqe=0 

CommRange=2000 

1 

Distributions 

Distributions 

1 

FinalLocation 

FiiitilLocation 

) 

Frequencies 


Level 
529-625 
632-134 
644-271 
Total 
N  Missing 

3  Levels 


Count  Prob 

1  0,14286 
4  0,57143 

2  0,28571 
7  1,00000 


Frequencies 


Level 
288-578 
377-838 
554-488 
588-492 
641-114 
Total 

N  Missing 

5  Levels 


Count 

2 


Prob 
0,33333 
1  0,16667 
1  0,16667 
1  0,16667 
1  0,16667 
6  1,00000 


Figure  19.  Distribution  of  the  final  locations  split  by  the  communication  range. 


The  author  re-ran  some  scenarios  and  noticed  that  these  final  locations  result  from 
terrain  issues.  Mostly,  the  terrain  around  the  last  location  is  not  smooth  enough  to  allow 
the  convoy  to  move  on. 

Due  to  the  fact  that  at  most  two  runs  per  design  point  are  terminated  by  the  time 
limit,  the  author  evaluates  their  occurrences  as  non-critical  and  leaves  them  within  the 
data  set. 
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2.  MOE  Selection 

In  this  first  approach,  the  author  decided  to  get  a  broad  overview  on  the 
simulation  outcome  by  looking  at  the  combined  correlation  matrix,  including  the  input 
factors  and  MOEs.  The  first  question  is  whether  communication  has  any  impact  on  the 
outcome.  In  this  experiment,  the  communication  range  has  two  levels,  0  and  2000,  which 
can  be  translated  as  having  and  not  having  communication.  In  the  former  case  the 
convoy  can  take  advantage  of  getting  perfect  information  without  latency. 

The  blue  box  in  Figure  20  indicates  that  having  communications  increases 
mission  length  (positively  correlated),  decreases  the  number  of  broken  blockades  and  the 
number  of  hits  (both  negatively  correlated).  The  way  communication  affects  the  outcome 
fulfills  the  author’s  expectations.  The  levels  do  not  show  a  strong  correlation  (near  -1  or 
1),  but  they  differ  noticeably  from  zero. 


Collections 


Blockade  A  Blockade  B  Blockade  C  CommRange  Mission  Length  Broken  Blockades  #HITS 


Blockade  A 

1 ,0000 

0,4956 

0,5271 

0,0000 

0,1133 

0,2424 

0,4205 

Blockade  B 

0,4956 

1 ,0000 

0,5271 

0,0000 

0,2141 

0,2224 

0,4381 

Blockade  C 

0,5271 

0,5271 

1 ,0000 

-0,0000 

0,2024 

0,2546 

0,3955 

CommRange 

0,0000 

0,0000 

-0,0000 

1 ,0000 

0,2088 

-0,1348 

-0,1980 

Mission  Length 

0,1133 

0,2141 

0,2024 

0,2088 

1 ,0000 

0,2291 

0,3915 

Broken  Blockades 

0,2424 

0,2224 

0,2546 

-0,1348 

0,2291 

1 ,0000 

0,5024 

#HITS 

0,4205 

0,4381 

0,3955 

-0,1980 

0,3915 

0,5024 

1 ,0000 

Figure  20.  This  correlation  matrix  shows  the  impact  of  having  communication  for  the 
MOEs  (blue  box)  and  the  correlation  among  the  MOEs  which  makes  a 
pre-selection  possible.  (Best  viewed  in  color) 


The  red  box  in  Figure  20  demonstrates  the  correlation  among  the  MOEs. 
Obviously,  the  three  MOEs  are  correlated.  The  correlation  between  “Broken  Blockades” 
and  “#Hits”  reaches  0.5.  This  is  meaningful  due  to  the  fact  that  fighting  through  the 
blockades  is  only  possible  for  the  convoy  within  a  25  meter  radius  of  the  blockade.  This 
is  the  range  in  which  the  blockade  has  its  highest  probability  of  hit.  The  other 
correlations,  including  with  mission  length,  are  clear,  but  not  so  strong.  However,  all 
correlations  are  positive.  This  means  that  examining  one  MOE  deeply  provides 


40 


information  about  the  other  two  MOEs.  In  this  case,  the  author’s  focus  lies  on  the 
number  of  hits  the  convoy  gets.  Nevertheless,  the  analysis  on  the  other  two  MOEs  is  also 
conducted.  The  results,  only  briefly  mentioned  here,  are  not  consistent.  In  a  leap  ahead, 
the  p-values  for  the  paired  t-test  on  the  two  samples  for  each  MOE  do  not  stay  below  the 
given  level  of  significance  of  0.05.  This  means  that  at  some  design  points  the  means  for 
one  MOE  with  and  without  communications  are  different,  but  they  are  not  statistically 
different.  There  remains  the  chance  that  both  samples  are  from  populations  with  the 
same  mean  at  the  given  design  point.  The  most  dangerous  elements  in  combat  situations 
are  the  bullets.  In  the  author’s  opinion,  the  number  of  hits  is  the  most  important  measure, 
which  is  also  the  only  statistically  consistent  MOE  in  this  simulation. 

3.  “Measuring”  the  Impact  of  Communications 

This  section  deals  with  the  question  of  how  well  the  model  behaves.  The 
experimental  setup  allows  comparing  two  runs  with  the  same  blockade  setting  and 
different  communication  parameter  settings.  Referring  to  the  MOE  selection,  the  author 
uses  only  the  number  of  hits  for  examining  the  impact  of  communication.  Within  JMP,  a 
data  table  is  created  that  contains  the  needed  parameter  infonnation  and  the  number  of 
hits  with  communication  and  the  number  of  hits  without  communication  in  one  row.  In 
the  next  step,  the  differences  between  the  number  of  hits  without  and  without 
communications  are  calculated.  A  negative  value  means  that  the  convoy  with 
communications  suffers  more  hits  than  the  one  without  communications.  In  addition, 
columns  are  attached  containing  binary  indicators  of  whether  the  runs  with 
communication  ended  with  a  lower  or  equal  number  of  hits.  Instead  of  regarding  each 
run,  a  summary  table  is  created  which  contains  the  parameter  setting,  the  mean  of  the 
differences,  and  their  standard  deviations  for  each  design  point. 

At  first,  the  variability  is  checked  with  a  box  plot  containing  all  design  points. 
This  is  done  in  Figure  21  below. 
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Variability  Chart  foi  Hit  Difference 


Figure  2 1 .  The  variability  for  each  design  is  demonstrated  in  this  range  plot. 


The  extracted  data  contains  a  lot  of  variability.  Especially,  the  large  negative 
amplitudes  are  an  indicator  that  the  situation  is  inherently  highly  variable.  Agent-based 
models  are  stochastic  models  and  therefore  they  incorporate  randomness.  Small  negative 
amplitudes  could  have  been  caused  by  this  effect.  All  25  means  are  positive  and  with 
more  blockades  showing  they  are  clearly  distinguishable  from  zero.  Under  the 
hypothesis  that  communications  have  no  effect  on  the  outcome,  the  chance  of  observing 
fewer  hits  (i.e.,  positive  median  hit  difference)  at  all  design  points,  has  the  probability  of 
(V2)-  (=3*10'  )  if  the  design  points  are  uncorrelated.  An  empirical  test  shows  that  the 
correlations  among  these  25  samples  are  consistent  with  correlations  among  25  normally 
distributed  random  samples.  Therefore,  on  the  whole,  communications  help  the  convoy, 
especially  with  increasing  difficulty  level.  Some  scenarios  were  re-run  in  order  to  find 
the  true  story  behind  these  large  negative  differences.  The  main  causes  are  summarized: 

•  The  convoy  with  communication  gets  stuck  in  a  position  where  it  is 
exposed  to  hostile  fire.  The  convoy  without  communications  does  not 
stop  due  to  its  different  desires  and  route. 

•  The  occurrence  of  an  unlucky  event  is  also  observed.  The  convoy  with 
perfect  knowledge  avoids  blockades  in  the  first  wave  but  it  is  caught  by  an 
ambush  in  the  second  wave  while  the  convoy  without  communications 
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gets  hits  but  luckily  its  later  route  is  safer  without  any  ambush  situation. 
Of  course,  this  could  happen  in  the  real  world. 

•  In  some  cases  the  convoy  with  additional  inorganic  information  does  not 
behave  like  a  human  being.  Driven  by  its  weights  it  ignores  the  grid 
friends’  and  helpers’  signals  and  runs  straight  into  the  blockade,  or,  even 
worse,  into  an  ambush  where  fire  hails  on  it. 

The  first  analysis  is  an  eye  opener  of  the  model’s  performance.  It  is  obviously  not 
perfect,  but  the  question  is  how  “good”  or  useful  is  it.  This  question  is  examined  by 
taking  a  closer  look  at  the  means.  The  means  and  the  95%-confidence  interval  for  all  25 
design  points  are  shown  graphically  in  Figure  22. 


Oveilny  Plot 


50- 


40- 


X  30- 


20- 


10- 


□  Lower  Bound 


x — Mean  (Difference)  □  Upper  Bound 


5 

6 

7 

8 

9 

10 
11 
12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 


0-0-1-(1) 
O-l-O-(l) 

l-O-O-(l) 
0-l-l-(2) 

l-0-l-(2) 

l-l-0-(2) 

1-1  - 1-(3) 

1- 3-l-(5) 

2- l-3-(6) 

3- l-2-(6) 

1- 2-4-(7) 

2- 3-2-(7) 

2- 4-1 -(7) 

4- l-2-(7) 

1- 4-3-(8) 

3- 3-3-(9) 

5- 2-2-(9) 

3- 2-5-(10) 

4- 2-4-(10) 

5- 4-l-(10) 

2- 5-4-( 1 1 ) 

3- 4-4-(ll) 

4- 3-5-(12) 

4- 5-3-(12) 

5- 5-5-(15) 


Figure  22.  The  means  of  the  differences  and  95%-CI  for  all  25  design  points 
(#combination).  The  blockade  setting  is  listed  beside  [A-B-C-(Total)]. 
(Best  viewed  in  color) 

The  graph  indicates  the  main  trend  that  an  increasing  number  of  blockades  results 
in  larger  differences.  The  first  box  shows  that  1  blocked  street  in  wave  A  (combination  3) 
has  a  larger  impact  on  the  difference  than  the  two  previous  ones  due  to  a  lack  of 
alternative  routes  at  the  starting  point.  The  drops  (11  and  15)  occur  because  the  number 
of  blocked  roads  (wave  A)  decreases  in  comparison  to  the  previous  design  points.  This 

indicates  that  the  number  of  blocked  streets  at  the  beginning  strongly  influences  the 
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outcome.  The  density  of  blockades  in  the  first  wave  is  higher.  In  order  to  illustrate  this 
result  the  same  plot  is  shown  after  reordering  the  design  points  within  the  number  of  total 
blockades  (see  Figure  23).  The  ordering  can  be  interpreted  as  difficulty  levels  within  one 
group  of  design  points  having  the  same  number  of  resulting  blockades.  Looking  at  the 
combinations  23  and  24,  it  can  be  concluded  that  the  setting  4-5-3  is  harder  than  4-3-5. 
The  argument  holds  that  the  blockade  density  in  wave  B  is  higher  than  in  wave  C. 
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□  Lower  Bound  x — Mean  (Difference)  a  Upper  Bound 
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Figure  23.  Reordered  plot  showing  the  means  of  the  differences  and  95%-CI  for  all 
25  design  points  (#combination).  Means  in  the  same  group  are  connected. 
The  blockade  setting  is  listed  beside  [A-B-C-(Total)].  (Best  viewed  in 
color) 


All  of  the  means  on  the  differences  are  positive.  Having  communications  reduces 
the  expected  number  of  hits.  If  not,  the  chance  of  seeing  this  is  a  rare  event.  This  implies 
that  the  model  does  a  “good”  job  on  average.  To  gain  more  insights,  the  proportions  of 
runs  which  do  better  with  communication  are  examined.  The  graph  in  Figure  24 
demonstrates  the  proportions  for  each  design  point.  The  ordering  of  the  design  points 
remains  as  found  in  the  previous  section. 
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Figure  24.  Comparison  between  runs  with  and  without  communications  showing  the 
percentage  of  how  many  runs  have  less  hits,  equal  number  of  hits  and  the 
combined  measure.  The  blockade  setting  is  listed  beside  [A-B-C-(Total)]. 
(Best  viewed  in  color) 

Two  main  trends  are  observable  in  Figure  24.  The  green  line  (□)  describes  the 
proportion  of  runs  with  the  same  outcome.  In  a  low  threat  scenario,  e.g.,  for  one 
blockade  model,  most  runs  have  the  same  outcome.  Increasing  the  threat  level  causes  the 
equal  outcomes  to  decrease.  On  the  other  hand,  the  proportion  of  runs  where 
communications  have  a  strictly  positive  influence  (0)  on  the  result  is  increasing  at  the 
same  time.  The  sum  of  both  (x)  has  been  chosen  as  a  measure  of  the  model  quality. 

Due  to  the  high  variability,  the  performance  drops  down  below  70%  and  stays  in 
this  region.  Two  dips  in  this  nearly  horizontal  line  are  remarkable — in  fact  the 
performance  decreases  below  60%.  The  points  have  one  common  pattern.  At  the 
beginning  the  convoy  experiences  a  low  intensity  phase  which  is  followed  by  a  massive 
appearance  of  blockades  in  the  wave  B. 

The  lack  of  performance  will  be  discussed  again  at  the  end  of  this  Chapter  when 
the  author  summarizes  the  findings  and  evaluates  the  quality  and  usefulness  of  the  model. 
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After  the  graphical  examination  of  the  data,  a  sequence  of  statistical  tools  is 
applied  to  analyze  whether  the  two  samples  (with  and  without  communication)  are  from 
different  populations.  The  test  procedure  is  conducted  on  one  design  point  for 
demonstration  purposes.  The  entire  results  are  listed  in  the  Appendix  Figures  37  to  43. 

The  distribution  of  the  hit  differences  are  created  for  each  design  point  in  JMP. 
As  mentioned  before,  the  paired  t-test  can  be  applied  even  when  the  underlying 
distribution  is  not  bell-shaped  normally  distributed.  A  large  sample  size  guarantees,  by 
the  Central  Limit  Theorem,  that  the  mean  is  roughly  normally  distributed.  This  allows 
using  standard  techniques  to  examine  the  mean. 

The  author  decided  to  check  on  normality  as  well  before  using  the  t-test. 
Therefore,  a  fitted  normal  distribution  provided  by  JMP  is  included  (see  Figure  25). 


Blockade  A=1.  Blockade  B=1.  Blockade  C=1 


Distributions 


- Normal(8, 29,1 8,9728) 


Figure  25.  The  distribution  of  the  hit  difference  for  a  specific  blockade  setting  and  its 
fitted  normal  distribution. 

The  distribution  seems  be  bell-shaped  around  the  mean,  but  the  asymmetric 
behavior  towards  the  edges  allows  the  conclusion  that  this  data  is  not  normally 
distributed.  To  confirm  this  observation  a  Quantile-Quantile  plot  (QQ-plot)  32  is 


32  Douglas  C.  Montgomery,  “Introduction  to  Linear  Regression  Analysis,”  3rd  Edition,  2001,  p.  192 
“Normal  Probability  plot. ...  If  the  sample  observations  are  in  fact  drawn  from  a  normal  population  with 
mean  p  and  standard  deviation  o.  the  points  should  fall  on  a  straight  line  with  slope  o  and  intercept  p.  Thus 
a  plot  for  which  the  points  fall  close  to  some  straight  line  suggests  that  the  assumption  of  a  normal 
population  distribution  is  plausible.” 
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examined.  Normally  distributed  data  should  show  up  along  a  straight  line.  In  a  parallel 
approach  the  Shapiro-Wilk-test  is  conducted  (See  Figure  26). 


Quantile  Plot 


Goo<lness-of-Fit  Test 

Shapiro-WlkW 
W 

0.833173 


Test 

Prob<W 

<.0001 


Figure  26.  Qantile-Quantile  plot  showing  that  the  distribution  is  definitely  not 
normal.  This  result  is  confirmed  by  the  Goodness-of-Fit  test  shown  in  the 
box. 


Obviously,  normality  cannot  be  assumed  looking  at  the  plot  and  the  low  p-value 
(Figure  26  red  box)  provided  by  the  Shapiro-Wilk-test.  But  the  absence  of  normality 
does  not  mean  that  no  further  analysis  is  possible.  The  t-test  has  robustness  against  non- 
normality,  as  mentioned  in  the  previous  section.  A  large  sample  size  is  necessary  to 
apply  it  and  get  reliable  results.  One  hundred  runs  per  design  point  guarantee  a 
sufficiently  large  sample  size.  Actually,  due  to  the  sample  size,  this  can  be  seen  as  a  z- 
test.  JMP  provides  a  GUI  to  conduct  the  t-  test.  The  hypothesis  under  which  this  paired  t- 
test  is  conducted  is  stated  in  the  following  way: 

Ha  •  M  l  Ht Difference  —  ^ 

^ a  *  ^ Hii Difference  ^ 

Because  of  the  author’s  intent  to  work  out  the  improvements  by  having  inorganic 
information,  a  one-sided  test  is  preferred  and  seems  to  be  more  appropriate  to  answer  the 
research  question.  The  result  provided  by  JMP  is  shown  in  Figure  27. 
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Test  Mean=value 

Hypothesized  Value 

0 

Actual  Estimate 

8,29 

df 

99 

Std  Dev 

18,9728 

t  Test 

Test  Statistic 

4,3694 

Prob  >  |t| 

<,0001 

|  Prob  >  t 

<,0001  | 

Prob  <  t 

1 ,0000 

Figure  27.  The  test  result  for  the  paired  t-test  demonstrates  that  communications 
reduces  the  number  of  hits. 

The  p-value  is  smaller  than  0.001.  This  implies  that  the  hypothesis  must  be  rejected. 
Actually,  communications  significantly  reduce  the  expected  number  of  hits  which  is 
observed  by  positive  differences.  The  paired  t-test  is  the  most  powerful  test  on  two 
sample  means.  This  sequence  of  tests  is  also  conducted  for  the  other  24  design  points 
and  is  shown  in  Appendix  Figures  37  to  43.  The  results  are  similar  to  the  explained  one. 

4.  Discussion  and  Conclusions 

Looking  at  the  convoy’s  behavior  with  and  without  communications  it  seems  that 
the  received  inorganic  information  has  an  impact  on  the  expected  number  of  hits.  Having 
support  by  a  sensor  grid  implies  a  reduced  number  of  hits  on  average.  The  statistical 
analysis  shows  that  for  all  design  points  both  samples  have  different  means.  The  model 
does  the  right  thing  on  average.  Its  weakness  is  expressed  in  the  proportion  of  runs 
where  communications  does  not  positively  influence  the  outcome.  On  average  30%  to 
40%  (maximum  44%)  of  the  runs  ended  with  a  less  hits  when  the  convoy  does  not  have  a 
sensor  grid  support. 

This  portion  sounds  high.  However,  human  decision  making  does  not  lead 
automatically  to  the  optimal  solution.  Sometimes  a  decision  has  to  be  made  within 
seconds.  The  convoy  leader  is  under  combat  stress  and  has  time  limitations  for  his 
decision.  In  situation  like  this  a  human  decision-maker  relies  on  heuristics  and  biases.33 
The  perceived  information  is  processed  and  one  or  more  hypotheses  are  generated  and 
tested.  If  one  seems  adequate  it  is  chosen  as  a  possible  course  of  action.  Furthermore, 

33  Christopher  Wickens,  “An  Introduction  To  Human  Factors  Engineering,”  2nd  Edition,  2004,  p.  162. 
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the  humans  tend  to  stick  with  their  first  hypothesis.  Therefore,  alternatives  are 
underutilized  and  rejected.  This  behavior  can  result  in  non-optimal  decisions.  In  some 
cases  they  are  fatal  and  lead  to  a  large  number  of  causalities.  Also,  just  by  chance  (i.e., 
bad  luck)  a  good  decision  can  lead  to  a  worse  outcome  than  a  bad  decision. 

The  author  is  aware  of  the  high  variability  incorporated.  The  chance  of  non- 
optimal  outcomes  in  a  decision-making  process  is  smaller  than  demonstrated  by  the 
model.  With  this  knowledge,  the  following  analysis  has  to  be  interpreted  with  caution 
and  the  resulting  numbers  have  to  seen  more  as  directions  than  precise  thresholds. 


E.  ANALYSIS  ON  COMMUNICATION  ASPECTS 

This  section  focuses  on  the  analysis  of  the  main  experiment  examining  the 
communication  factors.  The  post  processed  data  set  (162,500  rows)  is  large.  The 
variability  within  the  set  is  enormously  high.  Regression  analysis  on  the  original  data 
leads  to  small  R-square  values,  which  is  a  measure  how  much  variability  is  explained  by 
the  regression  model.  Therefore,  the  author  decided  to  use  the  summary  data.  The  data 
is  grouped  by  design  points.  The  means  and  standard  deviations  on  the  MOEs  are 
calculated  and  represented  in  columns.  The  data  set  still  contains  1,625  elements.  The 
author  excluded  the  factor  MaxAge  because  it  is  linearly  dependent  on  Latency.  A  pair 
of  linearly  dependent  regressors  causes  multicollinearity  in  the  regression  model;  this 
would  degrade  the  quality  of  the  regression  model. 

1.  Pre- Analysis 

This  section  focuses  on  two  pre-tests.  Similar  to  the  previously  conducted 
analysis,  the  runs  are  examined  in  the  case  where  the  convoy  did  not  accomplish  its 
mission.  Following  this,  the  correlation  among  the  MOEs  is  analyzed. 

a.  Unfinished  Runs 

The  unfinished  runs  are  extracted.  Their  distribution  is  shown  in  Figure 

28. 
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Figure  28.  The  distribution  of  the  final  location. 

The  distribution  chart  shows  that  24  locations  are  involved.  The  author  re¬ 
ran  some  of  them.  One  location  has  a  remarkable  count.  Location  644-271  (red  box) 
contains  88  runs  with  one  common  seed.  These  runs  are  unfinished  due  to  terrain  issues. 
Following  the  argumentation  in  section  Dl,  the  convoy  gets  stuck.  The  surrounding 
terrain  is  not  smooth  enough  to  allow  the  convoy  to  move  on. 

b.  MOE  Selection 

The  correlation  among  the  3  MOEs  is  checked  in  advance  in  order  to 
reduce  the  number  of  MOEs  to  be  tested.  The  correlation  matrix  below  (Figure  29) 
demonstrates  that  in  this  experiment  a  very  strong  positive  correlation  exists  among  all 
MOEs. 

Correlations 

Mean(Mission  Length  (real))  Mean(Blockades  Broken)  Mean(#HITS) 

Mean(Mission  Length  (real))  1,0000  0,6631  0,7190 

Mean(Blockades  Broken)  0,6631  1 ,0000  0,9431 

Mean(#HITS)  0,7190  0,9431  1,0000 

Figure  29.  The  correlation  matrix  shows  that  the  MOEs  are  strongly  correlated. 

In  comparison  to  the  previous  experiment,  these  correlations  are  powerful 
(near  1).  Being  consistent,  the  number  of  hits  is  chosen  as  the  main  MOE.  However,  in 
this  case,  the  author  did  not  do  any  analysis  on  the  rejected  MOEs. 

2.  Regression  Tree  with  Noise  Factors  and  Communication  Factors 
In  this  first  approach  a  regression  tree  is  used  to  gain  insights  into  the 
relationships  between  the  factors.  The  regression  tree  does  not  require  any  assumptions 
in  order  to  be  applied.  The  tree  structure  is  shown  in  Figure  30.  Additionally,  the  factors 
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involved  in  splitting  are  repeated  in  order  to  increase  the  readability.  In  this  step  the  split 
levels  and  resulting  group  means  and  standard  deviations  are  less  important  than  the 
factor  itself. 
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Figure  30.  Regression  tree  on  the  Mean(#Hit)  with  all  input  factors.  The  first  12 
steps  are  noise  factor  splits.  The  first  communication  factor  that  shows  up 
is  Latency  (see  red  box). 


The  presented  regression  tree  consists  of  13  splits.  At  the  top  the  “root”  of  the  tree 
contains  all  1625  observations.  The  first  12  partitions  involve  exclusively  the  noise 
factors.  It  can  be  concluded  that  the  blockades  (i.e.,  the  enemy  behavior),  are  the  major 
driving  factors  in  this  simulation.  Looking  at  the  sequences,  it  becomes  obvious  that  the 
blockades  have  a  non-linear  impact  on  the  outcome.  Also,  there  are  most  likely 
interactions  among  the  noise  factors.  The  first  communication  factor  appears  in  the  form 
of  Latency,  at  the  step  13.  This  indicates  the  importance  of  Latency  in  the  simulation. 
That  is,  the  convoy  needs  timely  information. 

These  findings  are  used  to  frame  the  next  analysis  part,  where  a  regression  model 
is  derived  that  includes  quadratic  terms  and  two-way  interactions  among  all  predictors. 

3.  Full  Regression  Model 

To  achieve  a  full  regression  model  including  all  statistically  significant  terms, 
JMP  is  used.  The  full  model  provides  insights  into  whether  the  underlying  assumption  of 
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additive  normally  distributed.  Another  test  will  be  conducted  to  reduce  multicollinearity, 
which  could  occur  because  of  the  correlation  among  the  noise  factors.  The  stepwise 
regression  function  in  JMP  allows  adding  and  removing  terms  at  each  step.  While  doing 
so,  the  order  of  the  factors  entering  the  model  can  be  observed.  This  is  helpful  to  see 
which  ones  are  more  important. 

The  ratio  of  the  variation  explained  by  the  model  to  the  total  variation  included  in 
the  data  is  called  R2,  the  coefficient  of  determination.34  The  model  shown  in  Figure  3 1 
has  a  high  R2  value  of  0.9752. 


Response  Mean(#HITS) 

Summary  of  Fit 

|RSquare  0,975171  | 

RSquare  Adj  0,97483 

Root  Mean  Square  Error  2,862534 

Mean  of  Response  28,54883 

Observations  (or  Sum  Wgts)  1 625 

Analysis  of  Vaiiance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

22 

515571 ,99 

23435,1 

2859,996 

Error 

1602 

13126,95 

8,2 

Prob  >  F 

C.  Total 

1624 

528698,94 

0,0000 

Paiametei  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

VIF 

Intercept 

.4,448744 

0,65584 

-6,78 

<,0001 

Blockades  A 

5,8087652 

0,06438 

90,23 

0,0000 

2,0857967 

Blockades  B 

4,6175575 

0,062567 

73,80 

0,0000 

1 ,9699865 

Blockades  C 

2,8279617 

0,066547 

42,50 

<,0001 

2,2286196 

Latency  (real) 

0,1175132 

0,004492 

26,16 

<,0001 

1 ,0003076 

Pause  (real) 
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0,0169 
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-0,88 
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-0,003143 
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<,0001 

3,3624782 

(Blockades  A-2,32)*(Blockades  C-2,32) 

0,1577005 

0,05329 

2,96 

0,0031 
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(Blockades  A-2,32)*(Latency  (real)-33,0154) 

0,0325866 
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<,0001 
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0,001 688 
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<,0001 
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Figure  3 1 .  The  full  regression  model. 


2 

The  fit  seems  to  be  good.  But  a  high  R~  value  does  not  automatically  guarantee  a 
good  model.  Over-fitting,  having  too  many  factors,  reduces  the  usability  for  predicting 
and  interpretability.  This  will  be  examined  in  the  next  section.  To  reach  this  high  R2 


34  Douglas  C.  Montgomery,  “Introduction  to  Linear  Regression  Analysis,”  3rd  Edition,  2001,  p.  39. 
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level,  22  predictors  plus  the  intercept  term  are  necessary.  The  model  still  includes  terms 
which  do  not  influence  the  outcome  significantly.  For  example,  the  senor  range  (Range) 
has  a  p-value  of  0.38.  This  would  normally  exclude  the  term;  but,  an  interaction  term 
involving  the  range  is  significantly  important  (see  connected  red  boxes).  Good  regression 
modeling  does  not  allow  taking  out  main  effects  that  are  involved  in  an  interaction. 

The  JMP  software  package  provides  a  test  on  multicollinearity.  The  Variance 
Inflation  Factor  (VIF)  is  automatically  calculated  for  all  regressors.  “The  VIF  for  each 
term  in  the  model  measures  the  combined  effect  of  the  dependences  among  the  regressors 
on  the  variance  of  the  term.”35  VIF  values  exceeding  5  or  10  are  an  indication  of 
multicollinearity.  VIF  also  shows  the  involved  factor.  In  this  case,  all  VIF  are  below  5, 
thanks  to  our  efficient  design.  Multicollinearity  seems  to  be  no  issue.  The  author  ran 
another  small  calculation  to  cross  check  this  result.  The  correlation  matrix  including  only 
the  regressors  is  used.36  The  determinant  of  the  correlation  matrix  has  the  range  between 
0  and  1.  The  result  for  a  matrix  including  at  least  one  pair  of  linearly  dependent 
regressors  is  0.  On  the  other  hand  orthogonal  regressors  are  indicated  by  a  value  of  1  for 
the  detenninant.  The  correlation  matrix  (see  Chapter  3  Figure  18)  has  a  determinant  of 
0.47.  The  loss  of  orthogonality  is  obvious,  but  the  distance  from  0  is  large  enough  to 
confirm  the  JMP  results. 

In  the  next  step,  the  error  term  is  examined.  The  differences  between  the  fitted 
regression  and  the  observed  values  are  called  residuals.  Their  distribution  is  shown  in 
Figure  32. 


35  Douglas  C.  Montgomery,  “Introduction  to  Linear  Regression  Analysis,”  3rd  Edition,  2001,  p.  337. 

36  Douglas  C.  Montgomery,  “Introduction  to  Linear  Regression  Analysis,”  3rd  Edition,  2001,  p.  343. 
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-Normal(7,6e-1 4,2,84308) 


Figure  32.  The  residual  distribution  with  fitted  normal  curve  and  the  result  of  the 
Shapiro-Wilk  W-test. 


The  residual  distribution  has  a  normal  bell  shape.  Looking  at  the  QQ-plot  above, 
the  endings  show  a  slightly  unusual  behavior.  Even  with  a  high  W  value  (near  1  means 
normality),  this  behavior  causes  the  rejection  of  the  hypothesis  that  the  residuals  are 
normally  distributed. 

Knowing  that  a  main  regression  testing  assumption  is  not  strictly  met,  the 
robustness  of  regression  against  slight  non-normality  is  used.  The  interpretation  of  the  p- 
values  whether  a  predictor  stays  in  the  model  or  not,  has  to  be  done  more  carefully  due  to 
the  slightly  non-normal  behavior  of  the  error  terms.  However,  this  does  not  impact  the 
significant  digits  of  the  p-values. 

4.  Reduced  Regression  Model 

The  section  deals  with  the  question  of  how  to  reduce  the  number  of  predictors 
while  still  maintaining  a  good  model  fit.  During  the  stepwise  regression  the  author  noted 
the  improvements  in  the  R  value  whenever  a  new  predictor  was  added.  The  graph  in 
Figure  33  shows  the  improvement  curve  as  a  function  of  added  regressors. 
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Figure  33.  R“  improvement  by  adding  a  new  predictor. 


Looking  at  the  chart,  it  is  remarkable  that  Blockades  A,  B  and  C  as  main  effects 
explain  95%  of  the  variation  in  the  data.  This  result  corresponds  to  the  regression  tree  in 
the  previous  section  where  the  first  13  steps  created  an  R“  of  about  95%.  Again,  Latency 
appears  as  the  first  communication  factor.  The  next  set  of  terms  are  the  interactions 
between  Blockades  A  and  latency  and  the  quadratic  terms  of  Latency,  Blockades  B,  A 
and  C.  With  the  8th  element  an  R2  of  97%  is  reached.  The  added  improvement  by  adding 
more  predictors  is  minimal.  Therefore,  the  author  decided  to  drop  the  elements  appearing 
after  the  8th  prdeictor  and  ignore  these  terms.  The  resulting  model  still  explains  97.1%  of 
the  variation.  The  reduced  model  is  shown  in  Figure  34. 
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Sumniaiy  of  Fit 

RSquare 

0,970951 

RSquare  Adj 

0,970807 

Root  Mean  Square  Error 
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Mean  of  Response 
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1625 

Analysis  of  Variance 


Response  Mean(#HITS) 


Paiametei  Estimates 

Term 
Intercept 
Blockades  A 
Blockades  B 
Blockades  C 
Latency  (real) 

(Blockades  A-2,32)*(Latency  (real)-33,0154) 
(Blockades  A-2,32)*(Blockades  A-2,32) 
(Blockades  B-2,32)*(Blockades  B-2,32) 

(Latency  (real)-33,0154)*(Latency  (real)-33,0154) 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

8 

513340,69 

64167,6 

6751 ,737 

Error 

1616 

15358,24 

9,5 

Prob  >  F 

C.  Total 

1624 

528698,94 

0,0000 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

-5,654431 

0,261101 

-21 ,66 

<,0001 

5,7354801 

0,065913 

87,02 

0,0000 

4,6237153 

0,063442 

72,88 

0,0000 

2,9598465 

0,064269 

46,05 

<,0001 

0,1175985 

0,004837 

24,31 

<,0001 

0,0371252 

0,003037 

12,23 

<,0001 

-0,261045 

0,03734 

-6,99 

<,0001 

0,3935432 

0,035732 

11,01 

<,0001 

-0,003664 

0,000343 

-10,67 

<,0001 

Figure  34.  Reduced  regression  model. 


The  interaction  term  (Blockades  A 515  Latency)  can  be  interpreted  as  both  factors 
driving  the  number  of  hits  up  additionally  when  they  are  at  high  settings  jointly,  and  vice 
versa.  The  results  show  again  that  latency  is  the  most  important  controllable  factor. 

The  model  reduction  eliminates  the  communication  factors  reliability,  capacity, 
update  frequency,  and  the  sensor  range.  Their  impacts  on  the  outcome  are  marginal  and 
can  be  neglected  in  comparison  to  latency  and  the  red  blockades.  The  scenario 
implemented  in  MANA  is  a  static  one.  The  changes  only  occur  at  3  points  in  time.  In  a 
more  realistic  scenario  modeling  more  dynamic  elements,  the  appearance  of  other 
communication  factors  in  a  regression  model  is  possible.  The  same  argument  will  be 
used  in  the  next  section,  where  thresholds  for  the  communication  factors  are  developed 
by  using  a  regression  tree. 

5.  Regression  Tree  for  Communication  Factors 

This  approach  identifies  thresholds  for  the  communication  factors.  A  regression 
tree  is  constructed  by  partitioning  exclusively  on  these  factors  (see  Figure  35). 
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Figure  35.  Regression  tree  for  the  communication  factors.  Only  the  left  part  of  the 
tree  structure  is  shown. 


The  discovery  tour  through  the  binary  tree  starts  at  the  root  with  all  1625 
observations.  In  this  case,  only  the  left  part  with  “Latency  smaller  28  seconds”  is  shown 
for  readability.  The  latency  can  also  be  expressed  in  a  length  measure  which  underlines 
the  importance.  The  convoy  runs  at  30  km/h  through  the  street  channels.  In  28  seconds 
it  covers  about  225  meters,  roughly  8  m/s.  Following  the  left  branch,  the  next  latency 
split  occurs  at  1 1  sec,  or  about  90  meters.  When  the  amount  of  time  between  updates  is 
smaller  than  22  seconds  (185  m)  an  improvement  is  noticeable.  Sensor  range,  larger  than 
389  m,  has  also  a  positive  influence  on  the  outcome.  This  chain  of  thresholds  is 
reasonable.  It  is  noted  again  that  the  scenario  is  static.  In  a  dynamic  scenario  these 
thresholds  are  likely  to  drop  significantly. 
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Following  the  branch,  “Latency  >  11”,  it  seems  that  a  lack  of  real-time  support 
can  be  compensated  somewhat  by  a  higher  reliability  and  capacity. 

6.  Discussion  and  Conclusions 

The  static  character  of  the  model  and  the  weakness  resulting  from  the  verification 
experiment  were  stated  repeatedly.  As  such,  the  findings  of  this  experiment  have  to  be 
read  as  trends  rather  than  as  specific  recommendations. 

The  analysis  shows  that  the  scenario  is  driven  primarily  by  the  enemy  behavior. 
The  blue  force’s  reaction  relies  heavily  on  precise  and  non-delayed  information.  Besides 
the  noise  factors,  latency  has  the  largest  impact  on  the  outcome.  The  other 
communication  factors  do  not  have  the  same  importance.  But,  the  model  does  not  allow 
the  conclusion  that  they  are  unimportant.  The  lack  of  a  dynamic  threat  in  the  scenario 
and  the  MANA  limitations  make  it  impossible  to  strictly  exclude  these  factors. 
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V.  FINAL  DISCUSSION  AND  RECOMMENDATIONS 


The  previous  chapter  has  revealed  that  MANA  can  produce  useful  insights.  But, 
the  author  is  not  totally  convinced  of  their  meaningfulness.  In  particular,  the  analysis  on 
the  model  shows  that  it  has  limitations.  However,  it  is  still  statistically  stable  on  average, 
as  observed  by  the  paired  t-test.  Due  to  these  limitations,  the  objective  of  exploring 
communication  aspects  cannot  be  completely  and  satisfactorily  answered.  Therefore,  the 
author  discusses  ideas  for  further  research  based  on  the  perspectives  gained  here.  These 
studies  are  difficult  when  it  comes  to  finding  solutions  to  combine  a  complex  terrain  and 
a  creative  movement  algorithm.  In  this  chapter,  improvements  and  alternative 
approaches  are  introduced  and  discussed.  First,  suggestions  for  improving  the  existing 
model  that  the  author  could  not  implement  due  to  time  constraints  are  discussed.  Then,  a 
short  general  excursion  is  made  into  network  analysis  in  order  to  show  another  potential 
decision  making  tool.  The  last  discussion  point  includes  a  totally  different  approach. 
Instead  of  using  a  time  step  agent-based  model,  the  potential  of  a  discrete  event 
simulation  is  explained. 


A.  IMPROVEMENTS  WITHIN  THE  EXISTING  MODEL 

The  replication  of  runs  uncovered  that  MANA’s  movement  algorithm  and  an 
urban  terrain  are  a  challenging  combination.  The  author  chose  to  let  the  convoy  use  the 
whole  width  of  the  streets.  This  led  to  a  situation  where  the  convoy  often  got  stuck  and 
chose  to  look  for  cover  against  enemy  fire  rather  than  escape.  As  a  result,  the  MOEs 
were  affected  negatively.  There  are  two  ways  to  overcome  this  bad  situation.  One 
solution  is  to  increase  the  convoy’s  desire  to  stay  away  from  walls  at  a  specified  distance. 
This  could  keep  the  convoy  in  the  middle  of  the  road.  The  alternative,  preferred  by  the 
author,  would  be  to  restrict  the  convoy  moves  to  a  line.  Imagine  a  railroad  track,  one 
narrow  guiding  line  in  the  middle  of  the  street  that  allows  the  convoy’s  movements.  The 
street  width  has  to  be  kept  large  to  allow  exposure  to  hostile  fire.  Of  course,  the  desire 
settings  have  to  be  adjusted.  The  repelling  character  of  blockade  or  ‘logic  helpers’  would 
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be  increased.  With  these  changes,  the  convoy  has  no  chance  to  circumvent  a  logic  helper 
and  can  not  ignore  its  warning  signal. 


B.  ALTERNATIVE  DECISION  TOOL 

The  MANA  agents  move  because  of  their  desires.  They  want  to  go  towards  a 
goal  or  stay  away  from  enemies.  A  penalty  function  delivers  the  movement  direction  by 
taking  the  minimum  penalty  associated  with  each  possible  choice.  All  of  these  decisions 
are  made  locally  at  each  time  step.  A  projection  into  future  steps  is  not  implemented, 
which  is  contrary  to  real  world  decision  making. 

The  modeled  scenario  is  a  classic  routing  problem  within  a  network  context.  The 
network  consists  of  nodes  and  arcs.  The  nodes  are  crossroads  and  the  arcs  are  connecting 
streets  between  two  nodes.  A  blockade  is  realized  as  a  high  arc  cost.  Network  analysis 
provides  several  algorithms37  to  find  the  best  path  through  such  a  network.  In  this  case, 
the  algorithm  would  deliver  the  safest  way — given  the  information  available.  The 
algorithm  results  in  an  ordered  list  of  nodes  the  convoy  has  to  visit.  The  nodes  can  be 
interpreted  as  waypoints  which  attract  the  convoy.  Only  the  next  waypoint  can  be  seen 
by  the  convoy.  Every  time  a  new  blockade  shows  up,  the  associated  arc  cost  is  increased 
and  the  list  recalculated.  Therefore,  movements  like  turning  around  and  going  back  to  a 
waypoint  already  visited  can  be  implemented. 

The  underlying  network  structure  should  be  implemented  as  a  third  map  in 
MANA.  The  convoy’s  decision  would  be  made  globally,  which  is  more  realistic  in  an 
urban  environment  when  non-local  information  is  available.  This  map  completes  the 
situational  awareness.  IT-SimBw,  the  simulation  software  developed  in  Gennany, 
includes  a  network  map.  Preliminary  results  reported  by  the  creators  look  promising. 


37  Ravindra  K.  Ahuja,  Thomas  L.  Magnanti,  and  James  B.  Orlin,  “Network  Flows:  Theory,  Algorithm, 
and  Applications,”  1993. 
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C.  DISCRETE  EVENT  SIMULATION 

The  simulation  community  has  a  divided  opinion  on  time  step  models,  like 
MANA.  Some  analysts  totally  deny  the  usefulness  of  time  step  models  and  prefer  a 
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discrete  event  simulation  (DES).  MANA  is  a  time  step  model,  even  though  events 
occur  and  the  agents  change  their  states.  The  author  will  briefly  outline  how  a  DES 
model  could  look  like. 

Picking  up  the  network  idea  from  the  previous  section,  the  street  map  can  be 
implemented  as  a  graph  with  nodes  and  arcs.  The  blockades  are  associated  as  cost  on  the 
arcs.  The  extra  cost  can  be  seen  as  additional  exposure  to  hostile  fire  and  be  expressed  as 
the  number  of  hits.  Each  node  knows  which  incoming/outgoing  arc  is  a  blocked  street 
and  how  many  hits  a  convoy  could  suffer  passing  this  node.  The  convoy  itself  carries 
lists  containing  the  network  infonnation,  that  is,  whether  the  streets  are  useable,  unknown 
or  blocked.  One  list  provides  the  estimated  number  of  hits  the  convoy  would  suffer  when 
passing  the  node  given  its  known  blockade  setting.  Another  list  provides  the  estimated 
time  it  would  take  to  traverse  an  arc.  These  databases  are  regularly  updated  due  to 
changes  in  the  convoy’s  situational  awareness.  The  visited  nodes  are  stored  in  a  stack. 
The  blockades  and  their  appearance  time  can  be  totally  randomized,  contrary  to  the 
scripting  of  MANA.  In  some  sense,  blockade  setting  can  be  modeled  as  a  smarter  way  of 
trying  to  set  up  ambushes. 

The  communication  link  between  a  sensor  and  the  convoy  can  be  modeled.  For 
example,  latency  could  be  achieved  by  entering  an  additional  delay  before  the  lists 
carried  by  the  convoy  are  updated.  The  reliability  could  be  implemented  by  the  decision 
of  whether  the  convoy’s  lists  are  renewed  or  not.  An  update  frequency  is  modeled  by 
periodically  scheduling  the  next  sensor  updates. 

In  an  event  driven  simulation,  the  convoy  makes  a  global  decision,  using  a 
shortest  path  algorithm,  regarding  which  route  to  take.  The  arrival  at  the  next  node  is 
scheduled  depending  on  the  arc  length  (traveling  time).  The  algorithm  tries  to  minimize 

38  Wikipedia.org  (November  2006),  Definition:  “In  discrete  event  simulation,  the  operation  of  a 
system  is  represented  as  a  chronological  sequence  of  events.  Each  event  occurs  at  an  instant  in  time  and 
marks  a  change  of  state  in  the  system.” 
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the  number  of  hits  while  getting  to  the  goal  in  a  reasonable  time.  The  convoy  listens  to 
the  sensor.  Every  time  its  graph  is  updated  a  new  path  decision  is  made — perhaps  after 
some  delay.  In  the  case  where  the  convoy  returns  to  a  prior  node  an  additional  delay  is 
activated,  perhaps  resulting  in  a  longer  exposure  to  fire.  On  its  way  through  the  streets 
the  convoy  collects  the  number  of  hits  stored  at  the  nodes,  or  incurs  extra  costs  by 
fighting  through  a  blockade. 

In  retrospect,  a  discrete  event  simulation  on  an  arc-node  structure  seems  to  be 
most  appropriate  approach  to  gain  insights  about  communication  aspects  of  convoys  in 
urban  terrain. 


62 


APPENDIX 


This  Appendix  contains  tables  and  figures  that  complete  the  experimental  design 
and  the  following  analysis.  The  complete  data  table  for  the  noise  factors  combining  a 
NOLH  design  and  8  additional  design  points  is  presented.  For  the  analysis  part,  the  Ruby 
code  is  inserted.  In  the  main  text,  only  one  design  point  was  explicitly  examined  in  order 
to  assess  the  model’s  quality.  The  complete  analysis  on  all  design  points  is  summarized 
in  this  Appendix. 


Super  agent  1 

Super  agent  2 

Super  agent  3 

Blockade  A 

Blockade  B 

Blockade  C 

Blockade 

total 

Low  Level 

7 

10 

13 

High  Level 

11 

14 

17 

Decimals 

0 

0 

0 

DP  1 

8 

14 

16 

4 

1 

2 

7 

DP  2 

7 

11 

17 

5 

4 

1 

10 

DP  3 

8 

12 

13 

4 

3 

5 

12 

DP  4 

8 

13 

14 

4 

2 

4 

10 

DP  5 

10 

14 

15 

2 

1 

3 

6 

DP  6 

11 

11 

15 

1 

4 

3 

8 

DP  7 

10 

11 

17 

2 

4 

1 

7 

DP  8 

9 

14 

16 

3 

1 

2 

6 

DP  9 

9 

12 

15 

3 

3 

3 

9 

DP  10 

10 

10 

14 

2 

5 

4 

11 

DP  11 

11 

13 

14 

1 

2 

4 

7 

DP  12 

11 

12 

17 

1 

3 

1 

5 

DP  13 

12 

12 

16 

0 

3 

2 

5 

DP  14 

8 

10 

15 

4 

5 

3 

12 

DP  15 

7 

13 

16 

5 

2 

2 

9 

DP  16 

9 

13 

13 

3 

2 

5 

10 

DP  17 

9 

11 

14 

3 

4 

4 

11 

DP  18 

11 

15 

18 

1 

0 

0 

1 

DP  19 

12 

14 

18 

0 

1 

0 

1 

DP  20 

12 

15 

17 

0 

0 

1 

1 

DP  21 

11 

14 

18 

1 

1 

0 

2 

DP  22 

11 

15 

17 

1 

0 

1 

2 

DP  23 

12 

14 

17 

0 

1 

1 

2 

DP  24 

11 

14 

17 

1 

1 

1 

3 

DP  25 

7 

10 

13 

5 

5 

5 

15 

Table  4.  (Appendix)  Complete  spreadsheet  for  the  noise  factors.  Design  points  (DP)  1 
to  17  are  the  NOLH  design  points.  18  to  25  are  additional  ones.  In  the  model 
the  super-agents  are  varied.  The  resulting  blockade  settings  are  calculated  in 
the  right  part. 
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run  =  0 

outputFile  =File.new("c:/thesis/sep27/HitsMissingDP27sep.txt","w") 
outputFile«"RUN, EXCURSION, #HITS\n" 
while  run  <  1 
excursion  =  0 
while  excursion  <100 

filename  =  "agt_end."+run.to_s+"."+excursion.to_s+".csv" 
path  =  "c:/thesis/sep27/missingdp/"+ filename 
readFile  =  File.open(path,"r") 
count  =  0 
while  count  <  7 
line  =  readFile.  gets 
values  =  line. chomp. split(/,/) 
count  =  count  +  1 
end 

hits  =  values  [10] 

xpos=values[4] 

ypos=values[5] 

outputFile«run.to_s+","+excursion.to_s+","+hits.to_s+","+xpos.to_s+"- 
"+ypos.to_s+"\n" 
excursion  =  excursion  +  1 
end 


Figure  36.  (Appendix)  Ruby-code  used  for  extracting  the  important  data  element  out 
of  the  provide  agent-end  data  fde.  MANA  writes  ordered  output  fdes. 
These  fdes  can  be  simply  accessed,  as  the  variable  filename  demonstrates. 
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Blockade  A=0.  Blockade  B=0.  Blockade  C=1 


Blockade  A=0.  Blockade  B=1.  Blockade  C=0 


Blockade  A=0.  Blockade  B=1.  Blockade  C=1 


Figure  37. 


(Appendix)  The  sequence  of  statistical  tests  conducted  to  assess  the  model 
quality.  Due  to  readability  the  25  design  points  are  split. 
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Blockade  A=1.  Blockade  B=2.  Blockade  C=4 


Distii  Initio  ns 


Hit  Difference 


• - 10— . 

^fF 

Tlw 

-100  -50  0 

50  100  150 

- Normal(8,39, 38,4358) 


Fitted  Normal 


Test  Mean=value 


100.0% 

maximum 

161 ,0 

Mean 

8,39 

99.5% 

161 ,0 

Std  Dev 

38,435804 

97.5% 

139,8 

Std  Err  Mean 

3,8435804 

90.0% 

43,4 

upper  95%  Mean 

16,016497 

75.0% 

quartile 

21 ,0 

lower  95%  Mean 

0,7635027 

50.0% 

median 

4,0 

N 

100 

25.0% 

quartile 

-3,0 

10.0% 

-24,0 

2.5% 

-80,4 

0.5% 

-116,0 

0.0% 

minimum 

-116,0 

Parameter  Estimates 

Type  Parameter 
Location  Mu 
Dispersion  Sigma 
Goodness-of-Fit  Test 
Shapiro-Wilk  W  Test 

W  Prob<W 
0,842549  <,0001 


Lower  95%  Upper  95% 
0,76350  16,01650 

33,74690  44,64992 


Hypothesized  Value  0 

Actual  Estimate  8,39 

df  99 

Std  Dev  38,4358 


t  Test 

Test  Statistic  2,1829 
Prob  >  |t|  0,0314 

Prob  >  t  0,0157 

Prob  <  t  0,9843 


Estimate 

8,39000 

38,43580 


Figure  38. 


(Appendix)  The  sequence  of  statistical  tests  conducted  to  assess  the  model 
quality.  Due  to  readability  the  25  design  points  are  split. 
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Blockade  A=1,  Blockade  B=3,  Blockade  C=1 
Distributions 
Hit  Difference 


Moments 


Fitted  Noimal 


Test  Mean=value 


-Normal(9,1 4,23,1 8) 


100.0%  maximum 
99.5% 

97.5% 

90.0% 

75.0% 

50.0% 

25.0% 

10.0% 

2.5% 

0.5% 

0.0% 


quartile 

median 

quartile 


116,0 

116,0 

66,4 

39,6 

22,0 

4,0 

-5,0 

-12,8 

-29,5 

-32,0 

-32,0 


Mean  9,1 4 

Std  Dev  23,180024 

Std  Err  Mean  2,3180024 
upper  95%  Mean  1 3,73942 
lower  95%  Mean  4,5405802 
N  100 


Paiametei  Estimates 


Type  Parameter 
Location  Mu 
Dispersion  Sigma 


Estimate  Lower  95%  Upper  95% 
9,14000  4,54058  13,73942 

23,18002  20,35222  26,92766 


Hypothesized  Value 
Actual  Estimate 
df 

Std  Dev 


0 

9,14 


Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 
0,894650  <,0001 


Test  Statistic 
Prob  >  |t| 
Prob  >  t 
Prob  <  t 


t  Test 
3,9431 
0,0002 
<,0001 
0,9999 


Blockade  A=1.  Blockade  B=4.  Blockade  C=3 


Distributions 


Hit  Difference 


-  Normal(1 1 ,37,42,534) 


Quantiles 

Z)l 

Moments 

EM 

100.0% 

maximum 

181 ,0 

Mean 

11,37 

99.5% 

181 ,0 

Std  Dev 

42,53404 

97.5% 

119,1 

Std  Err  Mean 

4,253404 

90.0% 

65,7 

upper  95%  Mean 

19,809676 

75.0% 

quartile 

24,0 

lower  95%  Mean 

2,9303237 

50.0% 

median 

4,0 

N 

100 

25.0% 

quartile 

-8,8 

10.0% 

-19,9 

2.5% 

-80,3 

0.5% 

-146,0 

0.0% 

minimum 

-146,0 

Fitted  Normal 


Test  Mean=value 


Parameter  Estimates 


Type  Parameter 
Location  Mu 
Dispersion  Sigma 
Goodness-of-Fit  Test 
Shapiro-Wilk  W  Test 

W  Prob<W 
0,887186  <,0001 


Estimate  Lower  95%  Upper  95% 
11,37000  2,93032  19,80968 

42 ,53404  37 ,3451 8  49 ,41 074 


Hypothesized  Value 
Actual  Estimate 
df 

Std  Dev 


Test  Statistic 
Prob  >  |t| 
Prob  >  t 
Prob  <  t 


t  Test 
2,6732 
0,0088 
0,0044 


Blockade  A=2.  Blockade  B=1.  Blockade  C=3 


Distributions 


Hit  Difference 


•  — m —  • 

.  r^f 

-50  0 

50  100  150 

Quantiles 

rj\ 

Moments 

JI 

100.0% 

maximum 

159,0 

Mean 

10,36 

99.5% 

159,0 

Std  Dev 

29,200982 

97.5% 

75,8 

Std  Err  Mean 

2,9200982 

90.0% 

43,0 

upper  95%  Mean 

16,154108 

75.0% 

quartile 

22,0 

lower  95%  Mean 

4,5658916 

50.0% 

median 

5,0 

N 

100 

25.0% 

quartile 

-1 ,8 

10.0% 

-16,5 

2.5% 

-46,4 

0.5% 

-70,0 

0.0% 

minimum 

-70,0 

Fitted  Normal 


Test  Mean=value 


Parameter  Estimates 


Type  Parameter 
Location  Mu 
Dispersion  Sigma 


Estimate  Lower  95%  Upper  95% 
10,36000  4,56589  16,15411 

29,20098  25,63866  33,92206 


Hypothesized  Value 
Actual  Estimate 


0 

10,36 


Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 
0,883189  <,0001 


Std  Dev 

t  Test 

Test  Statistic 

3,5478 

Prob  >  |t| 

0,0006 

Prob  >  t 

0,0003 

Prob<t 

0,9997 

-Normal(1 0,36 ,29,201) 


Figure  39. 


(Appendix)  The  sequence  of  statistical  tests  conducted  to  assess  the  model 
quality.  Due  to  readability  the  25  design  points  are  split. 
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Blockade  A=3.  Blockade  B=2.  Blockade  C=5 


Figure  40. 


(Appendix)  The  sequence  of  statistical  tests  conducted  to  assess  the  model 
quality.  Due  to  readability  the  25  design  points  are  split. 
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Blockade  A=3.  Blockade  B=3.  Blockade  C=3 
Distributions 


Hit  Difference 


— m— . 

r 

100  -50  0 

50  100  150  200 

- Normal(1 5,68,37,531 9) 


Quantiles 


100.0% 

maximum 

99.5% 

97.5% 

90.0% 

75.0% 

quartile 

50.0% 

median 

25.0% 

quartile 

10.0% 

2.5% 

0.5% 

0.0% 

minimum 

Moments 

203,0 

Mean 

203,0 

Std  Dev 

123,5 

Std  Err  Mean 

59,7 

upper  95%  Mean 

25,3 

lower  95%  Mean 

8,0 

N 

-4,8 

-14,6 

-55,0 

-76,0 

-76,0 

Fitted  Nonnal 


Test  Mean=value 


15,68 

37,531907 

3,7531907 

23,127145 

8,2328554 

100 


Parameter  Estimates 


Type  Parameter  Estimate  Lower  95%  Upper  95% 

Location  Mu  15,68000  8,23286  23,12714 

Dispersion  Sigma  37,53191  32,95327  43,59989 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 
0,837833  <,0001 


Hypothesized  Value 
Actual  Estimate 


df 

Std  Dev 

Test  Statistic 
Prob  >  |t| 
Prob  >  t 
Prob  <  t 


0 

15,68 
99 

37,5319 
t  Test 
4,1778 
<,0001 
<,0001 
1 ,0000 


Blockade  A=4.  Blockade  B=2.  Blockade  C=4 


Hit  Difference 


Quantiles 


Moments 


Fitted  Normal 


Test  Mean=value 


-  Normal(1 1 ,93,36,8303) 


100.0%  maximum 
99.5% 

97.5% 

90.0% 

75.0% 

50.0% 

25.0% 

10.0% 

2.5% 

0.5% 

0.0% 


quartile 

median 

quartile 


112,0 
112,0 
98,4 
58,9 
26,8 
7,0 
-5,8 
-23,6 
-71 ,4 
-139,0 
-139,0 


Mean  1 1 ,93 

Std  Dev  36,830284 

Std  Err  Mean  3,6830284 
upper  95%  Mean  1 9,237927 
lower  95%  Mean  4,6220727 
N  100 


Parameter  Estimates 


Type  Parameter 
Location  Mu 
Dispersion  Sigma 


Estimate  Lower  95%  Upper  95% 
11,93000  4,62207  19,23793 

36,83028  32,33724  42,78483 


Goodness-of-Fit  Test 

Shapiro-Wllk  W  Test 

W  Prob<W 
0,934972  <,0001 


Hypothesized  Value 
Actual  Estimate 


Test  Statistic 
Prob  >  |t| 
Prob  >  t 
Prob  <  t 


0 

11,93 
99 

36,8303 
t  Test 
3,2392 
0,0016 
0,0008 
0,9992 


Figure  41. 


(Appendix)  The  sequence  of  statistical  tests  conducted  to  assess  the  model 
quality.  Due  to  readability  the  25  design  points  are  split. 
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Blockade  A=5.  Blockade  B=4.  Blockade  C=1 


Disti  ihutions 


Hit  Diffeience 


-  Normal(22,56,44,9532) 


Quantiles 

II 

Moments 

Hilt 

100.0% 

maximum 

205,0 

Mean 

22,56 

99.5% 

205,0 

Std  Dev 

44,953246 

97.5% 

162,1 

Std  Err  Mean 

4,4953246 

90.0% 

77,9 

upper  95%  Mean 

31 ,479699 

75.0% 

quartile 

35,0 

lower  95%  Mean 

13,640301 

50.0% 

median 

12,0 

N 

100 

25.0% 

quartile 

-2,0 

10.0% 

-9,0 

2.5% 

-52,9 

0.5% 

-66,0 

0.0% 

minimum 

-66,0 

Fitted  Noimal 


Test  Mean=value 


Paiainetei  Estimates 


Type  Parameter 
Location  Mu 
Dispersion  Sigma 


Estimate 

22,56000 

44,95325 


Lower  95% 
13,64030 
39,46926 


Upper  95% 
31 ,47970 
52,22107 


Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob=W 
0,835160  <,0001 


Hypothesized  Value  0 

Actual  Estimate  22,56 

df  99 

Std  Dev  44,9532 

t  Test 


Test  Statistic 
Prob  >  |t| 
Prob  >  t 
Prob  <  t 


5,0185 
<,0001 
<,0001 
1 ,0000 


Figure  42. 


(Appendix)  The  sequence  of  statistical  tests  conducted  to  assess  the  model 
quality.  Due  to  readability  the  25  design  points  are  split. 
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Figure  43.  (Appendix)  The  sequence  of  statistical  tests  conducted  to  assess  the  model 
quality.  Due  to  readability  the  25  design  points  are  split. 
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